Olmo 3 — part 7/19

Cleaned-text code points [93008, 108755). Bibliography omitted here; source.md includes it.

We see in Table 6 that across all base model metrics, as well as in evaluations of subsequent SFT training, newer candidate mixes consistently improve performance. Notably, between Round 3 and Round 5 we also introduce our decontamination process, which means that the gains of Round 5 relative to Round 1 and Round 3 are likely underestimated in this table, given that only Round 5 reflects decontaminated data.

##### Performance shows substantial domain tradeoffs

Alongside our central integration tests, we also conduct exploratory 100B anneals with heavy skews toward particular domains, to better understand domain tradeoffs. We treat code/math/thinking capabilities as one domain group, and generative/QA capabilities as another domain group—and create modified mixes each prioritizing one of these groups while omitting the other. Our Gen-QA mix increases proportions of web, QA, and instruction data while omitting math, code, and thinking, and our math-code-thinking mix increases proportions of math, code, and thinking data while omitting QA and instruction data (but keeping web to avoid excessive skew away from pretraining distribution).

Table 7 shows results from these runs, compared against our final Round 5 midtraining mix. We see that training on our Gen-QA mix results in a substantial drop in math and code performance, while approximately matching the final mix in MC STEM \text{MC}_{\text{STEM}} , MC Non-STEM \text{MC}_{\textsc{Non-STEM}} , and GenQA performance. By contrast, in our math-code-thinking mix, math and code performance substantially exceeds that of our final mix—however, MC STEM \text{MC}_{\text{STEM}} , MC Non-STEM \text{MC}_{\textsc{Non-STEM}} , and GenQA performance take a notable hit.

These results indicate that there are real tradeoffs when skewing toward certain of these domains over others during midtraining. We see in particular that there is clear potential to further improve math and code performance by increasing weight of these domains in the mix—however, this comes at a significant cost to our MCQA and GenQA performance. Increasing weight on Gen-QA domains, on the other hand, yields minimal improvement on QA tasks, while predictably hurting math and code performance. Overall, these results suggest that our final midtraining mix strikes a healthy balance across these domains, avoiding too heavy of a domain skew and enabling strong final performance across metrics.

We also see these domain tradeoffs at the individual source level, observable in results from microanneals. Table 8 shows a microanneal comparison for the Reddit-to-Flashcards dataset, which relative to the web-only baseline yields improvement for multiple choice tasks, as well as a boost for certain code tasks, but results in some performance decrease in math and GenQA tasks. Conversely, in Table 9 we see that our novel synthetic reasoning data—meta-reasoning and program-verifiable reasoning—yields significant improvement in math and code tasks, but results in some performance drop on certain GenQA and MCQA tasks.

##### Thinking/instruct data benefits base performance

We also investigate the overall impact of inclusion of our post-training-oriented data—instruction and thinking trace data—through 100B integration tests on one of our intermediate midtraining mixes both with and without inclusion of these data subsets (holding total mix tokens constant). Table 10 shows base eval performance after each of these training runs—we see that the mix that includes these post-training elements performs better on every base eval measure. This suggests that although individual sources and domains present performance tradeoffs, the inclusion of these cross-domain post-training data types in aggregate is consistently beneficial, and this benefit begins even before post-training.

##### Leave special tokens for SFT stage

To inform our formatting for instruction datasets, we also conduct an investigation to determine the impacts of inclusion or omission of special chat tokens such as <|im_start|> and <|im_end|> in our midtraining data. We test this via microanneals on the Tulu3-SFT data, comparing versions with and without these tokens. Experiments show that when training on data containing chat templates and special tokens, models consistently output these special tokens at inference time, resulting in evaluation scores that are dramatically reduced (e.g. GSM8K drops from 49.43 to 0, and CruxEval drops from 32.89 to 18.91). Further analysis highlights that simply including a chat template, with ordinary text in place of special tokens, did not produce the same performance drop (46.02 on GSM8K and 29.65 on CruxEval), suggesting that this disruption in model behavior is not due to inclusion of a chat template more generally, but is rather due specifically to the introduction of special tokens to the embedding vocabulary when they have not been seen in pretraining.

Though the degradation in model evaluation scores can be attributed primarily to disruption in answer parsing, these results highlight the broader issue that inclusion of these tokens at midtraining time results in emission of these tokens by the base model at inference time. Since this is an undesirable behavior, we ultimately remove both the chat template and special tokens from our instruct data, and revert to simple newline-based formatting.

##### Extent and impact of decontamination are variable

Figure 12 shows the top ten midtraining data sources containing the most occurrences of benchmark contamination. We find that much of the contamination occurs in existing datasets such as Flan and Nemotron. Not all contamination was subtle—we found many templated contamination instances, in which fields from benchmarks were exactly matched, with templated content inserted between them. Furthermore, many of these were not isolated instances, but complete validation or test splits. For instance, Flan is constructed from templates on benchmark data, and can include validation data that is used for model development decisions since test sets are hidden (e.g., DROP).

Performance is sometimes, but not always, inflated by contamination. We investigate this by comparing our final decontaminated 100B anneal with a matched 100B anneal using the non-decontaminated data versions. Figure 12 also shows the extent to which benchmark performance after midtraining drops when contamination is removed ( Perf Δ \Delta ). Some differences are substantial—such as validation or test performance changes in DROP, Minerva, SQuAD. Note that we remove contamination of all splits for all benchmarks, such as for DROP removing over 60,000 training examples from sources such as Flan. So performance differences may indicate that decontamination is preventing memorization or also removing in-distribution training examples. We remove all splits because some of our development benchmarks increase sample size by evaluating on train and held out splits (Figure 12 , right) and several of these also show performance overestimation with contamination of any of the evaluated benchmark splits. However, other benchmarks do not show inflated performance, despite contamination: we see that DeepSeek LeetCode performance is close to 0 with or without contamination, and SQuAD under the easier MC metric is saturated in either case. Finally, similarly to reports from Marin 32B ( Hall et al., 2025 ) , we find that despite the fact that our decontamination procedure detected complete leakage of GSM8K in our data, this does not result in better performance with the contaminated data. Instead we see that performance is in fact better with the decontaminated data, a phenomenon that the Marin authors explain occurs due to the contaminated formatting not matching the evaluated format. 23 23 23 This discussion was disseminated on social media .

##### Model souping can improve midtraining performance

For Olmo 3 Base 32B, we observe noteworthy performance improvement from merging two independent midtraining runs with differing seeds. Relative to the individual midtraining runs, the merged model yields nearly a full point of improvement in the MC STEM \text{MC}_{\text{STEM}} task cluster, 0.4 improvement in the GenQA task cluster, and in the Math task cluster result in improvements of 2.9 and 1.6 relative to the first and second midtraining runs, respectively. Other noteworthy improvements include approximately 1 point of improvement in MMLU, and 5 and 2 points of improvement in GSM Symbolic relative to the first and second runs. For this reason, we select the merged model as our final midtrained 32B checkpoint. 24 24 24 Initial experimentation for the 7B model did not show similar gains from model merging, so the 7B midtrained checkpoint is the result of a single run.

### 3.6 Stage 3: Long-context Extension

A crucial ability for modern language models is the capacity to operate over long sequences. This capability is necessary to process the long inputs required by many real-world tasks. Moreover, generating long sequences of intermediate tokens is a common technique to achieve test-time scaling ( Muennighoff et al., 2025b ) . In this section, we provide an overview of the methodology we used to scale Olmo 3 ’s context window from 8,192 to 65,536 tokens. We also describe Dolma 3 Longmino Mix , a high-quality dataset of both naturally-occurring and synthetically-augmented long texts. Dolma 3 Longmino Mix consists of over 600 billion tokens ; statistics in Table 11 .

##### Long-context extension strategy

Because training with long sequence lengths is computationally costly, most language models are pretrained with shorter sequences and extended only in a later stage of model development. During the extension phase, models are trained on longer documents and the hyperparameters of positional embeddings are typically adjusted to ease positional generalization.

##### High variance in open-model recipes

The recipes for performing this long-context extension vary dramatically between models. The context extension phase for many language models ranges from hundreds of billions (SmolLM3: 100B, Bakouch et al. 2025 ; GLM 4.5: 100B, GLM-4.5 Team et al. 2025 ; DeepSeek V3: 123B, DeepSeek-AI et al. 2025 ; Apertus: 225B, Apertus Team 2025 ) to almost one trillion tokens (Kimi K2: 400B, Kimi Team et al. 2025 ; Llama 3.1: 800B, Grattafiori et al. 2024 ; DeepSeek V3.1: 840B, DeepSeek-AI 2025 ). However, there are outliers: AFM ( Goddard, 2025 ) and Nemotron Nano 2 ( NVIDIA et al., 2025 ) both use fewer than 20 billion tokens to extend to 64K and 128K, respectively. Standalone extension recipes have also been proposed, many emphasizing token efficiency. For instance, ProLong ( Gao et al., 2025 ) uses 20B tokens drawn from books and code, whereas LongAttn ( Wu et al., 2025b ) constructs a 5B-token corpus using self-attention scores from existing language models to select documents exhibiting long-range dependencies. Another key point of divergence across model families is when in the development pipeline the extension is performed: Llama 3.1 models apply long-context extension prior to midtraining, Qwen 2.5 and 3 perform it afterwards, and GLM 4.5 applies extension only after supervised finetuning.

##### Olmo 3 long-context recipe

To extend Olmo 3 ’s context, we use long documents from the olmOCR science PDFs pool (Section § 3.6.1 ) with additional filtering and synthetic data augmentation applied (Section § 3.6.2 ). We call this collection Dolma 3 Longmino Pool . We mix 34% long-context data with 66% high-quality short-context data sampled from Dolma 3 Dolmino Mix , and train using this mix for an additional 50B tokens for Olmo 3 7B and 100B tokens for Olmo 3 32B, as described in Section § 3.6.3 . During long-context extension, we apply YaRN ( Peng et al., 2023 ) to full attention layers, and do not adjust positional embeddings on sliding-window attention layers; we use document packing and inter-document masking (Section § 3.6.3 ). We summarize the key aspects of our recipe in Figure 13 . While developing this recipe, we carefully analyze and isolate architectural design decisions that have profound impact on long-context performance; our investigation is presented in Bertsch et al. (2026) .

##### Overall results

We evaluate our context-extended models on two popular long-context benchmarks. RULER ( Hsieh et al., 2024 ) is a benchmark of synthetic long-context tasks including challenging variations of the Needle-in-a-Haystack task ( Nelson et al., 2024 ) and simple aggregation tasks that require counting over inputs; we use RULER as the primary metric to guide our long-context recipe development. HELMET ( Yen et al., 2025 ) is a suite of long-context benchmarks across a diverse set of task types, including retrieval, in-context learning, and summarization tasks, which we evaluate on to represent more general long-context capabilities. We keep HELMET as an unseen evaluation suite and test our final checkpoints on it. 25 25 25 There is some overlap between RULER and HELMET, so this is not a perfect held-out suite; however, the overlapping subsets are generally the easier ones where models trivially achieve near-perfect performance. See Appendix A.8 for details. We report results in Table 12 .

#### 3.6.1 Sourcing Long Context Data

##### olmOCR science PDFs

The backbone of our long-context data pool is scientific PDFs scraped from the web and processed by olmOCR . 26 26 26 See Section 3.4.2 for more details on the preprocessing of this data. Figure 14 describes the distribution by topic in each length bucked shown in Table 11 .

##### Data filtering

We filter this data using gzip compressibility as a metric. gzip has been used for text classification ( Jiang et al., 2022 ) and as a feature in fine-grained scaling laws ( Pandey, 2024 ) . We use gzip for data filtering by excluding the extremes: removing the 20% of text that is most compressible and the 20% of text that is least compressible.

We also consider applying filters based on LongPpl ( Fang et al., 2025b ) , which identifies tokens that rely moston long-range dependencies by measuring, for each token, the change in perplexity under an existing long-context model when additional preceding context is provided. We compute LongPpl over 10B tokens of Dolma 3 Longmino Mix using Gemma 3 4B ( Gemma 3 Team, 2025 ) as the reference model, and comparing contextualization using 4K or 128K context windows. We use the same threshold as Fang et al. (2025b) for determining whether a token is a “key” token that requires long context dependencies.

We compute two statistics over each document: the fraction of tokens marked as key tokens, and the spread of key tokens across the document (which we compute as the standard deviation of key token locations, which are measured relative to the document length). In a sweep of experiments, we consider excluding the bottom 20% of documents with the least key tokens or lowest spread, and excluding both the top and bottom 20% as outliers; none of these possibilities outperform the gzip filter, so we do not use this for the final run.

#### 3.6.2 Experiments with Synthetic Augmentation

A common use case for extended-context language models is information extraction and synthesis over long inputs ( Bai et al., 2024 ; Bai et al., 2025 ) . However, most long documents do not provide supervision for such tasks. Directly inspired by CLIPPER ( Pham et al., 2025 ) , we modify a portion of our science PDF pool by injecting synthetically generated aggregation tasks at randomly sampled intervals. Our approach also shares similarities with Qwen 2.5 1M ( Yang et al., 2025b ) .

##### Generation pipeline