Olmo 3

arxiv:2512.13961

Original source · Paper JSON

LLMs: architectures, context, training · 2025-12-15

Cites (15) · Cited by (9)

Complete cleaned Markdown · Source Markdown including bibliography

Read in parts

1. Part 1
2. Part 2
3. Part 3
4. Part 4
5. Part 5
6. Part 6
7. Part 7
8. Part 8
9. Part 9
10. Part 10
11. Part 11
12. Part 12
13. Part 13
14. Part 14
15. Part 15
16. Part 16
17. Part 17
18. Part 18
19. Part 19

Contents

• Olmo 3
• Abstract
• 1 Introduction
• 2 Model Flow for Olmo 3
• 2.1 Base Model Training
• Evaluation
• Data curriculum
• Open artifacts
• 2.2 Post-training
• 2.3 Results
• 2.4 Costs
• 3 Olmo 3 Base
• 3.1 Main Results for Olmo 3 Base
• 3.2 Modeling and Architecture
• Architecture
• Training
• Tokenizer
• 3.3 Experimental Design and Evaluation
• 3.3.1 Clustering Tasks
• 3.3.2 Scaling analysis
• 3.3.3 Signal-to-Noise Analysis
• 3.3.4 OlmoBaseEval
• 3.4 Stage 1: Pretraining
• 3.4.1 Preparing our Web Data Pool
• Text extraction
• Heuristic filtering
• Deduplication
• Topic and quality classification
• Final web data pool
• 3.4.2 Preparing our olmOCR science PDFs Data Pool
• olmOCR text extraction
• Deduplication
• PII filtering
• Heuristic filtering
• 3.4.3 Preparing Code, Math, and other sources
• Code
• Math
• Other
• 3.4.4 Sampling and Mixing over Data Pools
• Constrained data mixing
• Quality-aware upsampling
• Evaluation during pretraining
• 3.5 Stage 2: Midtraining
• 3.5.1 Methodological framework
• Targeted capability boosts
• Integration tests
• 3.5.2 Capability Improvements for Final Data Mix
• Math capabilities
• Code capabilities
• QA and knowledge access capabilities
• Cross-Capability instruction data
• Cross-capability thinking traces
• High quality web and PDF data
• 3.5.3 Decontamination
• Method and tooling
• 3.5.4 Key findings
• Candidate mix quality improves over time
• Performance shows substantial domain tradeoffs
• Thinking/instruct data benefits base performance
• Leave special tokens for SFT stage
• Extent and impact of decontamination are variable
• Model souping can improve midtraining performance
• 3.6 Stage 3: Long-context Extension
• Long-context extension strategy
• High variance in open-model recipes
• Olmo 3 long-context recipe
• Overall results
• 3.6.1 Sourcing Long Context Data
• olmOCR science PDFs
• Data filtering
• 3.6.2 Experiments with Synthetic Augmentation
• Generation pipeline
• Synthetic aggregation tasks
• 3.6.3 Choosing Data Mix and Token Budget
• Interleaving long- and short-context data
• Longer extension helps
• 3.6.4 Curating a Training Recipe for Extension
• RoPE extension
• Document packing
• Intra-document masking
• LC training infrastructure
• Model souping
• 3.7 Base Model Results
• 4 Olmo 3 Think
• 4.1 Main Results for Olmo 3 Think
• 4.1.1 Evaluation Details
• 4.1.2 Main Results
• 4.2 Supervised Finetuning with Dolci Think SFT
• 4.2.1 Dolci Think SFT : Data Curation
• Step 1: sourcing prompts and generating reasoning traces
• Step 2: filtering
• Step 3: data mixing
• Step 4: decontamination
• 4.2.2 Training
• 4.3 Preference Tuning with Delta Learning
• Delta Learning
• 4.3.1 Dolci Think -DPO: Preference Data Creation
• Step 1: sourcing prompts and contrastive completions
• Step 2: filtering
• Step 3: mixing
• 4.3.2 Training
• 4.4 Reinforcement Learning with OlmoRL : The Cherry on Top
• 4.4.1 OlmoRL Algorithmic Details
• OlmoRL formulation
• Verifiers
• 4.4.2 Dolci-Think-RL : Curating a State-of-the-art RLVR Dataset
• Step 1: sourcing prompts
• Step 2: offline difficulty filtering
• Step 3: data mixing
• 4.4.3 OlmoRL Infrastructure in Open Instruct
• Fully asynchronous training
• Continuous batching
• Active sampling
• Inflight updates
• Better threading and engineering
• 4.5 Key Findings
• DPO yields gains where SFT on the same data cannot
• DPO and SFT both benefit from RL, but DPO remains a better starting point
• Rewards steadily increase across all domains during RL
• Mixing RL data from varied domains can prevent over-optimization
• Mixing data yields lower train reward, but not lower downstream performance
• Continuous batching and inflight updates are crucial to training speed
• OlmoRL shows significant improvement in precise instruction following
• 5 Olmo 3 Instruct
• 5.1 Main Results for Olmo 3 Instruct
• 5.2 Supervised Finetuning with Dolci Instruct SFT
• 5.2.1 Function-calling Training Data
• Trajectories with real interactions
• Trajectories with simulated interactions
• Balancing function diversity with interaction complexity
• Unified data format
• Evaluating function calling
• 5.2.2 Curating Dolci Instruct SFT
• Step 1. Sourcing Prompts and Completions
• Step 2: Filtering & Mixing
• Starting from Olmo 3 Think SFT
• 5.3 Preference Tuning with Dolci Instruct DPO
• 5.3.1 Preference Signals
• Delta-learning heuristic pairs
• Delta-aware GPT-judged pairs
• Multi-turn preferences
• Controlling length bias
• 5.3.2 Prompt Mixing
• 5.3.3 Training
• 5.4 Reinforcement Learning with Dolci Instruct -RL
• 5.4.1 Training
• 5.5 Key Findings
• Starting from the Olmo 3 Think SFT is helpful
• High contrast in preference pairs drives DPO improvements
• Combining different preference signals improves overall performance
• The ideal amount of preference data depends on the downstream task
• Concise, usable model outputs from preference tuning can boost RL performance
• Need for tools
• 6 Olmo 3 RL-Zero
• 6.1 Reinforcement Learning From Base with Dolci RL-Zero
• Data
• Prompt and eval template
• RL algorithm
• 6.2 Key Findings
• Olmo 3 RL-Zero can strongly improve on reasoning
• Olmo 3 RL-Zero mix can benchmark challenges in multi-objective RL
• Olmo 3 RL-Zero can benchmark reasoning data mixes in midtraining
• Active sampling stabilizes training
• Eval decontamination is verified via spurious rewards
• Appendix A Appendix
• Author Contributions
• Acknowledgments
• A.1 Base Model Additional Training Details
• A.2 Base Model Additional Data Details: Pretraining
• A.2.1 CommonCrawl
• A.2.2 Deduplication
• Exact Deduplication
• MinHash Fuzzy deduplication
• Suffix Array deduplication
• A.2.3 Topic Classification
• A.2.4 CommonCrawl Mixing
• Problem formulation
• Parameterizing the solution space
• Implementation details
• A.2.5 Validating Quality Upsampling and Mixing
• Targeted mixing
• Quality-aware upsampling
• Reconciling upsampling and mixing
• A.3 Base Model Additional Data Details: Midtraining
• A.3.1 Math Capabilities
• TinyMATH
• CraneMath
• MegaMatt
• OMR Rewrites
• Key Findings and Results
• A.3.2 Code Capabilities
• CraneCode
• A.3.3 Thinking Capabilities
• Meta-reasoning
• Existing thinking traces
• A.4 Base Model Additional Evaluation Details
• Expanding OLMES tasks
• A.4.1 Base Evaluation suites
• Base Easy suite
• Base Main suite
• Base Chat suite
• Base Long-Context suite
• Base Held-out Suite
• A.4.2 New Evaluation Benchmarks
• Basic Skills
• Gen2MC
• Masked perplexity
• A.5 Base Model Additional Decontamination Details
• A.5.1 Definitions and Preliminaries
• A.5.2 Example of Contamination
• A.5.3 Eval Normalization
• A.5.4 Decon Implementation
• Detecting contamination
• Cluster discovery
• IDF-weighted overlap
• Cluster match length decay
• Cluster discovery threshold
• Non-comparative
• Inverted index
• Hot n-grams
• A.5.5 Scoring System
• Length penalty
• Confidence adjusted weight
• Base scores
• Answer proximity
• Passage proximity
• A.6 Post-Training Additional Training Details
• A.6.1 Supervised Finetuning Details
• Using OLMo-core infrastructure for SFT Training
• A.6.2 Preference Tuning Details
• Training Settings
• A.6.3 Reinforcement Learning Details
• A.6.4 RL-Zero Details
• A.7 Post-Training Additional Data Details
• A.7.1 Filtering for Dolci Think -SFT
• A.7.2 Tool-use data
• Additional details about the Science QA dataset
• Additional details about the Web Search QA dataset
• Additional details about simulated interaction trajectories
• A.7.3 Coding Data Synthesis Pipeline
• A.7.4 Dolci Instruct DPO Details
• DPO prompt mixing
• Model pool for LLM-judged pairs
• A.8 Post-Training Additional Evaluation Details
• A.8.1 General Evaluation Settings
• Is AlpacaEval useful?
• A.8.2 Safety Evaluations Overview
• Development safety evaluations
• Unseen safety evaluations
• Averaging and reported metrics
• Instructions for reporting errors