Olmo 3 — part 3/19
Cleaned-text code points [29482, 45474). Bibliography omitted here; source.md includes it.
Fully-open Models Open-weight Models Olmo 3 7B Marin 8B Apertus 8B Gaperon 8B OLMo 2 7B Qwen3 8B Nemo. Nano 9B Gemma 2 9B Qwen 2.5 7B Llama 3.1 8B Granite 3.3 8B MiMo 7B OlmoBaseEval Math 54.7 39.6 29.2 16.9 41.7 67.2 49.8 48.8 60.7 36.9 41.5 54.3 GSM8k 75.5 60.9 48.2 30.0 67.1 84.5 82.3 68.5 79.9 56.4 61.0 74.3 GSM Symbolic 48.6 33.6 26.3 12.5 38.8 65.4 62.7 45.1 56.2 35.1 35.5 53.3 MATH 40.0 24.3 13.1 8.2 19.1 51.6 4.5 32.9 45.9 19.2 27.9 35.2 OlmoBaseEval Code 30.7 21.4 19.0 16.1 10.4 46.1 43.1 30.2 41.0 21.2 18.0 35.7 BigCodeBench 34.1 21.5 20.9 13.0 8.8 42.5 43.2 30.9 39.7 30.7 0.4 38.3 HumanEval 49.1 31.6 21.6 24.5 16.3 71.7 71.7 40.0 66.1 40.4 0.0 57.0 DeepSeek LeetCode 1.4 0.5 0.6 0.0 0.2 8.3 6.8 1.9 5.1 0.1 0.0 1.2 DS 1000 20.2 16.5 11.8 9.1 10.1 33.1 30.3 23.4 35.2 22.2 22.6 28.1 MBPP 43.6 36.5 33.5 29.3 21.2 66.2 62.3 49.1 55.4 12.1 48.5 48.3 MultiPL HumanEval 28.7 15.6 15.5 12.1 4.2 52.3 40.0 27.9 40.3 14.5 22.3 34.5 MultiPL MBPPP 38.2 27.6 29.2 24.6 12.2 48.4 47.5 38.2 45.4 28.3 32.3 42.5 OlmoBaseEval MC STEM \textbf{{OlmoBaseEval} \penalty\ MC}_{\textbf{\penalty\ STEM}} 66.4 68.1 66.3 58.0 64.6 78.8 73.5 72.8 74.7 69.0 65.0 71.6 ARC MC 89.2 89.2 87.9 77.2 85.7 95.4 94.1 92.7 93.4 86.4 86.2 91.7 MMLU STEM 59.7 58.1 52.4 43.1 53.2 76.7 71.1 62.8 67.6 55.7 55.6 63.5 MedMCQA MC 48.3 52.7 51.7 44.5 49.2 63.5 54.5 58.9 60.3 56.5 49.6 56.2 MedQA MC 41.8 47.3 47.6 36.8 43.8 62.1 53.5 55.4 56.6 53.7 43.0 53.0 SciQ MC 92.8 93.2 91.9 88.4 90.9 96.1 94.3 94.4 95.4 92.7 90.8 93.5 OlmoBaseEval MC Non-STEM \textbf{{OlmoBaseEval} \penalty\ MC}_{\textbf{\penalty\ Non-STEM}} 78.2 78.8 74.2 65.0 75.2 84.8 81.3 81.3 82.9 76.1 76.9 80.5 MMLU Humanities 68.9 71.4 67.8 59.5 67.9 78.6 78.0 74.5 76.2 70.1 67.6 73.6 MMLU Social Sci. 75.0 77.4 74.7 60.8 73.1 84.8 82.2 82.9 83.0 75.5 71.8 80.8 MMLU Other 66.9 68.3 66.1 57.2 65.2 76.8 73.8 74.2 74.4 69.1 64.5 72.7 CSQA MC 75.3 75.3 72.1 65.5 72.0 84.1 74.4 75.3 85.0 72.9 82.3 76.1 PiQA MC 80.2 85.7 80.5 71.6 80.1 89.9 86.0 85.7 88.5 78.3 81.5 87.2 SocialIQA MC 80.3 79.8 76.3 73.4 77.5 83.3 78.7 80.3 82.9 77.0 83.1 80.7 CoQA Gen2MC MC 92.5 86.2 82.8 59.7 85.0 93.7 92.2 92.7 93.5 89.9 87.6 91.4 DROP Gen2MC MC 67.3 63.7 47.5 44.8 55.6 78.3 70.0 65.8 69.1 53.3 55.0 64.1 Jeopardy Gen2MC MC 86.9 90.8 90.3 83.2 89.5 92.3 90.7 92.8 92.1 88.9 88.4 89.5 NaturalQs Gen2MC MC 69.4 71.5 66.7 51.3 66.3 74.1 71.1 72.5 70.5 68.0 69.2 72.2 SQuAD Gen2MC MC 96.9 96.5 91.3 87.7 95.3 97.5 97.4 97.3 96.4 94.4 94.5 96.7 OlmoBaseEval GenQA 72.5 75.9 69.0 63.3 72.4 71.1 71.8 75.6 67.5 73.1 67.8 71.4 HellaSwag RC 77.7 84.0 81.0 73.9 82.2 80.5 80.2 81.8 81.0 81.5 83.7 80.6 Winogrande RC 85.7 88.6 85.8 76.4 87.4 86.4 86.2 88.8 86.0 87.3 89.4 86.5 Lambada 68.9 73.9 70.9 67.0 70.5 73.0 67.9 76.3 70.3 75.5 76.0 73.1 Basic Skills 89.5 85.6 83.8 80.5 82.2 93.5 91.4 89.3 91.4 88.0 88.7 89.7 DROP 71.5 73.0 37.1 54.9 61.5 57.2 71.4 68.2 56.7 59.5 38.4 69.3 Jeopardy 60.4 72.7 70.1 55.5 70.8 65.1 64.9 75.1 63.0 70.9 69.7 65.6 NaturalQs 32.6 42.6 35.0 28.8 37.4 33.8 31.2 40.4 31.2 36.7 37.0 33.1 SQuAD 93.5 93.4 89.6 86.0 91.5 89.2 92.3 88.8 87.0 89.2 89.6 90.3 CoQA 72.8 69.5 67.4 46.7 68.3 61.6 60.4 71.5 40.5 69.0 37.8 54.4 OlmoBaseEval HeldOut LBPP 17.1 5.8 7.1 4.7 3.1 25.7 31.7 12.4 22.1 9.1 18.5 21.5 BBH 63.5 55.6 48.1 38.4 49.6 76.5 77.0 68.8 54.7 63.0 61.5 75.1 MMLU Pro MC 37.3 38.8 33.9 20.8 33.1 50.3 50.2 44.7 48.1 37.4 33.9 44.3 Deepmind Math 23.7 20.2 17.1 34.1 16.2 47.7 31.4 23.0 32.8 24.1 32.2 25.4
### 3.2 Modeling and Architecture
Olmo 3 modeling and training largely follows that of OLMo 2 . We focus this section on the key differences and refer to the appendix for further details.
##### Architecture
We adopt a decoder-only transformer architecture based on Vaswani et al. (2017) . Details of the architecture are presented in Table 33 in Appendix A.2 . Compared to OLMo 2 :
• We train with a context window of 8192 tokens (increased from 4096 tokens for OLMo 2 ) during pretraining and midtraining stages.
• To support scalable pretraining at longer sequence lengths, and to keep inference costs manageable, we introduce a sliding window attention (SWA) pattern ( Beltagy et al., 2020 ) in which each token can attend to previous tokens in a window of size 4096. We add SWA at three out of every four layers, and ensure that the last layer always uses full attention.
##### Training
Olmo 3 Base is trained using the OLMo-core 4 4 4 Further details and code: github.com/allenai/OLMo-core codebase. With this stack, we train the 7B model at 7700 tokens per second per GPU and the 32B model at 1960 tokens per second per GPU at a sequence length of 8192, using bfloat16 precision throughout. This corresponds to roughly 43% and 41% MFU, respectively. We achieve this performance by combining PyTorch’s built-in torch.compile() , custom kernels for operations such as attention ( Dao, 2024 ) and the language modeling head ( Hsu et al., 2025 ) , asynchronous and batched gathering of metrics, and asynchronous checkpoint writing, among other optimizations.
OLMo-core supports pretraining, midtraining, long-context extension, and SFT, along with auxiliary tools for checkpoint conversion to and from Hugging Face Transformers format and for merging model checkpoints. Support for DPO and RL is planned but not yet complete.
Hyperparameters for training Olmo 3 Base 7B and 32B are presented in Table 35 in Appendix A.2 . As in OLMo 2 , we train in stages defined by the data curriculum and learning rate schedule (see Appendix Table 35 for details). Infrastructure and distributed training configurations for each stage are summarized in Appendix Table 34 .
##### Tokenizer
We process data for each stage using the same tokenizer as OLMo 2 , which is derived from OpenAI’s cl100k ( OpenAI, 2023a ; OpenAI, 2023b ) .
### 3.3 Experimental Design and Evaluation
Model development requires many iterative data and training decisions. However, benchmarks are not perfect decision-making tools: different evaluations are only sensitive for making development decisions across specific ranges of scale and capability ( Magnusson et al., 2025 ) . Models trained at small compute scales are known to exhibit random-chance performance on math, code, and multiple-choice question answering (MCQA) tasks ( Wei et al., 2022 ; Gu et al., 2024b ) , and benchmark noise can reduce the ability to trust small differences in scores ( Heineman et al., 2025 ) . To address these problems, we develop OlmoBaseEval , a collection of benchmark suites to support decision-making during base model development. OlmoBaseEval features the following improvements: • We aggregate scores over task clusters that group benchmarks by assessed capability (Section § 3.3.1 ),
• We develop proxy metrics for evaluating small-scale models by identifying when capabilities “emerge” during training (Section § 3.3.2 ), and
• We improve the overall signal-to-noise ratio by evaluating more examples from noisy tasks or even removing them entirely (Section § 3.3.3 ).
We start by targeting a high coverage of capabilities; we select benchmarks to prioritize science knowledge, medical/lab knowledge, math, and code tasks. Because our data interventions are targeted to a core capability rather than a specific benchmark (e.g., “Code” rather than “DS-1000”), we group tasks into clusters , where we expect the benchmarks within a cluster to behave similarly to particular data changes. To handle evaluation of models trained using small compute budgets (e.g., up to our largest experiment scale of 1B parameters at 100B tokens), we perform a scaling analysis to determine which benchmarks show signal at a small scale and find proxy metrics which we use to make decisions. Finally, we analyze the signal-to-noise ratio of each benchmark—we select benchmark metrics to improve SNR, remove benchmarks that were too noisy for making decisions, and move benchmarks out of the average if the noise of one particular benchmark dominated the aggregate scores.
#### 3.3.1 Clustering Tasks
To handle the large number of tasks, we cluster similar tasks into macro-averages. We aim for task clusters to match the granularity at which we perform data interventions, and for tasks within each cluster to behave similarly. Our clustering procedure requires a process to determine the similarity of two evaluations—we do this by collecting a pool of 23K benchmark scores from 70 external, open-weight models.
Using our dataset of evaluation results, we assume that two benchmarks evaluate similar constructs if they rank models similarly. We perform hierarchical clustering using Ward’s variance-minimization ( Ward Jr, 1963 ) , which iteratively merges evaluation scores to minimize the variance of scores between benchmarks within a cluster. Figure 5 shows the result of the clustering procedure, where we manually select a threshold to balance the amount and granularity of clusters. Importantly, we do not use the exact result of the clustering procedure—we manually move a few tasks to ensure the format of the task is the same within each cluster (e.g., tasks requiring code execution all occur in the same cluster). The resulting task clusters are: MC STEM \text{{MC}}_{\text{{STEM}}} , MC Non-STEM \text{{MC}}_{\textsc{{Non-STEM}}} , GenQA , Math , Code , and Code FIM .
#### 3.3.2 Scaling analysis
We evaluate open-weight models across compute scales from 10 18 10^{18} to 10 25 10^{25} training FLOPs to determine the compute scale at which particular metrics and tasks are useful for development decisions. On some evaluation benchmarks, it is too difficult to see signal when training models at small scales ( Wei et al., 2022 ) , and other benchmarks ‘saturate’ near the labeling error of the benchmark ( Vendrow et al., 2025 ) . However, while many tasks appear emergent, continuous proxy metrics have been shown to be a better decision-making tool for model performance before we exit the noise floor ( Schaeffer et al., 2023 ; Huang et al., 2024b ; Magnusson et al., 2025 ) . We propose a Base Easy task suite which measures bits-per-byte (BPB) over tasks from the Base Main suite that have gold labels or human-written answers, calculated as the negative log-likelihood of the answer divided by the number of UTF-8 bytes in the answer string, as described in Gao et al. (2020) .
We evaluate on the suite of 25 OLMo 2 scaling law models from Bhagia et al. (2024) to understand the scaling behavior in the low-compute regime, and 70 open-weight models to understand scaling behavior in the high-compute regime. Figure 6 shows the scaling behavior for our resulting Base Main benchmarks. For each task family, the Base Easy task suite shows signal at the small data ablation scale, and the Base Main task suites were not saturated at the large scale, leaving headroom for data experiments in midtraining.
#### 3.3.3 Signal-to-Noise Analysis
When reporting a macro-average, we aim to exclude tasks from each cluster that were too noisy to be helpful for development. We calculate the signal-to-noise ratio of each benchmark following the method from Heineman et al. (2025) , where we evaluate the final 50 checkpoints of OLMo 2 13B training, and 10 external base models trained at roughly the same compute scale ( 4 ⋅ 10 23 4\cdot 10^{23} FLOPs). From our findings, we transition from using 1K instance subsets to full evaluation sets when available. We remove some benchmarks from our evaluation suite entirely, particularly binary benchmarks such as BoolQ ( Clark et al., 2019 ) , as we found that models usually oscillate between predicting the majority and minority class.
We repeat the same analysis for midtraining, instead using intermediate checkpoints from 5 preliminary pretraining runs. One important finding was to separate some benchmarks from the macro-average, like CruxEval ( Gu et al., 2024a ) , which measures a relevant and unique capability (code input/output prediction) but would introduce too much noise into the macro-average. We show an example of the SNR of three individual benchmarks compared to the base main task averages across intermediate checkpoints during midtraining in Figure 7 .
#### 3.3.4 OlmoBaseEval
The resulting OlmoBaseEval consists of a Base Easy suite for making development decisions using small compute budgets (e.g., less than 1B parameters) and a Base Main suite for development decisions for the final pretraining run and midtraining. We provide detail on the Chat suite later in § 4.1 . OlmoBaseEval contains 43 tasks, which is over 4 times more benchmarks than OLMo 2 —including tracking math and code benchmarks in pretraining. To prevent overfitting on the development suite, we include a Held-out set of 4 benchmarks—MMLU Pro, DeepMind Math, LBPP, and BBH—each benchmark matching one broad capability we target during pretraining.
The suite includes four new benchmarks: BasicSkills , a set of 6 tasks to isolate the development of skills during pretraining (e.g., basic arithmetic, reasoning, and coding); Gen2MC , a multiple-choice version of 5 short-form generative tasks; MT MBPP , a translated BPB set for MBPP in 17 code languages; and Masked Perplexity , a new evaluation method applying token masking and calculating perplexity only on tokens that are difficult to learn. We evaluate with masked perplexity using UltraChat and WildChat, which provides a wide coverage of real user interaction evaluation in pretraining. Additional design and implementation details for OlmoBaseEval are included in Appendix A.4 .
### 3.4 Stage 1: Pretraining
We first train Olmo 3 Base on Dolma 3 Mix , our 6T token pretraining data mix. While Dolma 3 Mix is comprised of largely the same types of data sources used in other open pretraining recipes ( Soldaini et al., 2024 ; Bakouch et al., 2025 ; OLMo et al., 2024 ) , we demonstrate three key novelties: • New tooling for fast and scalable global deduplication at the trillion-token scale;
• Two new methods for optimizing selection of training tokens: token-constrained mixing and quality-aware upsampling;
• A novel source of academic PDFs— olmOCR science PDFs—converted to linearized plain text using olmOCR (Section § 3.4.2 ) ( Poznanski et al., 2025a ) .
Table 4 summarizes our data sources, pool sizes, and final training mix. 5 5 5 The training mixes that we release represent reconstructions of the data sampled during our actual training runs. Tokens included in these reconstructions represent all of the tokens trained on for the training run, while included documents represent a union of all unique documents that contributed at least one token during training. As developing a base model is the most compute-intensive part of our development process, requiring training over trillions of tokens and consuming over 90% of overall compute, we adhere to two major principles to guide our data strategy: • We consider a source of data for pretraining if it has potential to yield enough tokens to impact model capabilities at pretraining scale. Valuable data sources that are small may not be impactful in pretraining and are better reserved for midtraining.
• While we embrace exploration of structured “task” data (e.g. QA pairs, chat instances) for training base models, we reserve their use only for later stages of midtraining (Section § 3.5 ) and long-context extension (Section § 3.6 ). Task data often does not meet the pool size needed to impact our pretraining stage, even with synthetic generation, and task data also tends to have an outsized impact on evaluation results, potentially confounding data ablations for other sources.
Figure 8 summarizes the pipeline steps for creating Dolma 3 Mix pretraining data. We describe them in more detail in the remainder of this section.
#### 3.4.1 Preparing our Web Data Pool
We took the following steps to curate pretraining data from CommonCrawl ( Common Crawl Foundation, ) , which constituted the majority of our pretraining corpus.
##### Text extraction