Olmo 3 — part 6/19

Cleaned-text code points [77384, 93008). Bibliography omitted here; source.md includes it.

##### Code capabilities

Our efforts to improve code capabilities include two major threads: 1) curation of higher-quality general code data, and 2) introduction of fill-in-the-middle (FIM) code capabilities. The top-performing datasets included in the final mix are the following:

• Stack-Edu (FIM) We include a modified version of Stack-Edu, in which 50% of documents reflect fill-in-the-middle (FIM) transformation via the infilling procedure from StarCoder2 ( Lozhkov et al., 2024 ) . This transformation splits code documents into prefix, middle, and suffix segments in order to train on prediction of the concealed middle segment. To further improve the quality of this code data, we apply quality filtering by performing reservoir sampling and bucketing of documents based on educational value score, 16 16 16 For educational value score we use language-specific classifiers provided developed for Hugging Face SmolLM model series, e.g. huggingface.co/HuggingFaceTB/stack-edu-classifier-php . followed by weighted random sampling of the upper 20% of buckets from each language subset. Microanneals validate that this quality filtering combined with the sampling procedure improves code benchmark performance over both the natural distribution of Stack-Edu and more naive sampling procedures such as sampling the top document per language based on classifier score.

• CraneCode As with our math datasets, we find strong performance from the SwallowCode dataset, and generate a permissively-licensed recreation for use in our midtraining. Like Fujii et al. (2025) , we source data from the Python subset of the-stack-v2-smol 17 17 17 huggingface.co/datasets/bigcode/the-stack-v2-train-smol-ids , released by Lozhkov et al. (2024) . , then filter for syntax errors and filter based on linter outputs. Then, we apply the SwallowCode two-stage rewriting pipeline, with one stage to augment style, and another to optimize the code itself. This yields 18.8B tokens of high-quality python code. In a microanneal using 5B tokens of high-quality data, CraneCode results in a lift in HumanEval of 5.0 points relative to pre-anneal baseline, compared to the 10.3 seen for SwallowCode. When using a larger microanneal with 12.5B tokens of CraneCode, the lift in HumanEval improves to 13.5.

##### QA and knowledge access capabilities

We target improvements in question-answering and general knowledge access capabilities through synthesis of two novel datasets focused on particular QA capabilities, as well as inclusion of high-quality existing QA data. The final datasets included for these capabilities are the following:

• Reddit-to-Flashcards We synthesize this dataset in response to the need to handle diverse content categories and question structures in multiple-choice QA tasks. We first identify a subset of academically-relevant subreddits, and then use GPT 4o-mini to rewrite submission-comment pairs from those subreddits into multiple-choice QA pairs. We use seven task formats to increase diversity. Microanneals show that inclusion of 5B tokens of this data in a 10B-token microanneal resulted in over 2 points of improvement in the MC Non-STEM \text{MC}_{\text{Non-STEM}} task cluster—relative to a 10B-token web-only baseline microanneal—with 3 points of improvement in MMLU.

• Wiki-to-RCQA We synthesize this dataset in response to the need for improvements in passage-based reading comprehension QA. We collect Wikipedia passages and prompted Qwen2.5 32B Instruct to generate QA pairs based on these passages, meeting a range of constraints inspired by instructions given to annotators of reading comprehension QA datasets. Microanneals show that 4.2B tokens of this data in a 10B microanneal results in nearly 2 points of improvement in the GenQA task cluster relative to a 10B web-only baseline, with improvements focused on the DROP, SQuAD and CoQA reading comprehension QA benchmarks.

• Nemotron We include the “diverse QA pairs” synth subset of the Nemotron CC dataset ( Su et al., 2025a ) , as, in microanneals, it improved GenQA tasks by 1.5 points, MC Non-STEM \text{MC}_{\text{Non-STEM}} by 1.9 points, and it had equal MC STEM \text{MC}_{\text{STEM}} performance compared to a microanneal run of web documents from the top quality (5%) bucket. All other Nemotron synth subsets (“distill”, “extract knowledge”, “knowledge list”, and “wrap medium”) performed worse than natural data, so we did not use them.

##### Cross-Capability instruction data

To lay the groundwork for post-training, we include cross-domain instruction datasets to prime models for instruction-tuning.

• Tulu3 SFT data We sample instruction data from the SFT set from Tülu 3. Compared to dataset released by Lambert et al. (2024) , we lightly process these data as follows: 1) we use an expanded set of examples that were created and subsequently filtered out for the final Tülu 3 data, 2) instead of relying on post-train syntax, such as <|im_start|> and <|im_end|> , we concatenate messages using double newlines. We choose this format, rather than using special tokens after microanneal experiments comparing them. More details are provided in see discussion of special tokens in Section § 3.5.4 .

• Flan Through microanneals, we also find the Flan dataset ( Wei et al., 2021 ; Longpre et al., 2023 ) improves performance in QA tasks, and as a result included a subset of the Flan dataset in the final mix. We use same subset and preprocessing from OLMo 2 ( OLMo et al., 2024 ) .

##### Cross-capability thinking traces

We also curate a diverse collection of thinking traces across a variety of domains to lay the foundation for Olmo 3 Think and Olmo 3 RL-Zero . This includes two new synthetic datasets, as well as rewritten and filtered versions of existing thinking trace datasets. • Meta-reasoning The first of the two new datasets introduced in this work; we crate it to target seven core cognitive capabilities from Kargupta et al. (2025) that are foundational to mathematical and programming expertise: self-awareness ( Toy et al., 2024 ; Callaway et al., 2022 ) , evaluation ( Fleming and Daw, 2017 ) , goal management ( Ackerman and Thompson, 2017 ; Griffiths et al., 2019 ) , hierarchical organization ( Haupt, 2018 ) , backward chaining ( Olieslagers et al., 2024 ) , backtracking ( Joyce, 2009 ) , and conceptual reasoning ( Markovits et al., 2015 ) . These categories are inspired by work suggesting that meta-reasoning capabilities in base models may be associated with superior reinforcement learning trajectories ( Kargupta et al., 2025 ; Gandhi et al., 2025 ) . We express these capabilities into tasks 18 18 18 See Appendix Tables 43 and 44 for list of tasks, and github.com/allenai/dolma3/tree/main/datasets/dolma3_dolmino_mix/meta-reasoning for the prompts. that require levering meta-reasoning, such as backtracking from an answer back to its original math problem, or debugging a program. To generate our meta-reasoning data for each of these tasks, we synthetically augmented existing math ( Luo et al., 2025a ; Moshkov et al., 2025 ) and code ( Li et al., 2023a ; Hendrycks et al., 2021a ; Ahmad et al., 2025 ) problems with detailed annotations such as ‘problem classification’, ‘difficulty analysis’, ‘solution approaches’, ‘common pitfalls’, and ‘verification methods’, modeled after the Pandalla-Math dataset. 19 19 19 huggingface.co/datasets/pandalla/pandalla-math-dataset-v1.0 Using these annotations as foundation, we prompt GPT-4.1 and o4-mini to generate thinking traces for each capability-targeted task. Microanneals show that inclusion of this data results in substantial improvements to math and coding tasks, resulting in approximately 14 points of boost—relative to a strong math/code baseline microanneal—in Minerva Math, and 14 and 20 points of boost on Codex HumanEval and MBPP benchmarks, respectively.

• Program-verifiable data Our second new synthetic reasoning dataset consists of program-verifiable tasks ( Zeng et al., 2025b ) for which we can use a Python program to deterministically verify whether an answer to a problem is correct. Solving these problems naturally requires a wide range of meta-reasoning strategies that are well-suited to be learned during midtraining. We 1) programmatically generate these problems, 2) distill thinking traces from GPT-4.1 and o4-mini models, and 3) finally filter those for correctness using an output verifier (Python programs). Microanneals show that including about 250M verifiable data tokens (in a 5B microanneal) led to 1-2 points of improvement on math and code tasks, including GSM8K and MBPP, relative to a math/code microanneal baseline.

• OMR rewrite full-thoughts We also consider 9 different versions of rewriting 20 20 20 Documentation for this approach, including all prompts, is available at github.com/allenai/dolma3/datasets/dolma3_dolmino_mix/open_math_reasoning_rewrites . of the OpenMathReasoning dataset ( Moshkov et al., 2025 ) , and find top performance for what we call the Full-Thoughts rewrite. This is a light rewrite of the OpenMathReasoning dataset, instructing GPT-4.1 to edit items for clarity, flow, and formatting (e.g., converting to LaTeX) while preserving all reasoning, explanations, and thoughts of the original. In microanneals, training on all 850M OMR Full-Thoughts tokens and an equal amount of web text, we see a lift of 5.5 points in the MATH benchmark and a 8.4 lift in GSM8K.

• Existing thinking traces We also draw on a variety of existing synthetic thinking trace datasets, to which we apply a range of filtering steps to reduce noise and increase quality. These sources have coverage over a broad variety of domains, including math, code, natural sciences, social sciences, humanities, and puzzles. These datasets are listed in Table 5 , and more details are provided in Appendix A.3 . Microanneals show that inclusion of these datasets yielded improvements especially in math and code domains, with improvements of up to 8 points in GSM8K, and approximately 2 points in HumanEval and MBPP, relative to a math/code microanneal baseline.

Table 10 provides further results showing the impacts of inclusion of instruction and thinking data in our midtraining mix, at the level of full integration tests.

##### High quality web and PDF data

Finally, we include three types of web / pretraining data to avoid skewing too far from the pretraining distribution.

• Stage 1 web data We sample documents from the top two quality buckets (top 10% quality). We sample according to natural distribution, not the optimal ratio described in Appendix § A.2.4 . In tests, the optimal ratio from the pretrain stage results in no improvement over natural distribution; since it introduce additional implementation complexity, we abandon it for the midtraining stage.

• Stage 1 olmOCR science PDFs From our PDF documents (Section § 3.4.2 ) we create a further filtered version, which we use both for midtraining and for long-context extension. Instead of discussing details here, the reader will have to hold their breath till Section § 3.6.1 . This creates tension in the manuscript, giving them something to look forward to.

• Stem-heavy crawl We also create a separate high-quality web collection, crawled between September 12, 2024 and June 3rd, 2025 using our in-house crawler. The crawler ingested scientific, educational, and general domains based on domain-level seeds sourced from manual lists of websites deemed high value. We use same crawling policy described as olmOCR science PDFs (Section § 3.4.2 ). Through microanneal experiments, we choose to filter this set using the quality classifier introduced in Section § 3.4.1 ; in detail, we use a threshold score of 0.6, which corresponds to the top 2.83 % 2.83\% of the data we crawled, and would make put these sources in the top 0.79 % 0.79\% of web data in the Dolma 3 pool. Relative to a web-only baseline, our crawled data yields an improvement of approximately 2 points each for MC Non-STEM \text{MC}_{\text{Non-STEM}} , MC STEM \text{MC}_{\text{STEM}} , and Math subsets of OlmoBaseEval .

#### 3.5.3 Decontamination

Earlier Olmo models have enabled research on benchmark contamination in base model training, such as decontamination of perplexity evaluations ( Magnusson et al., 2024 ) or measuring the impact of quality filters on evaluation leakage ( Godey et al., 2025 ) . In Olmo 3 midtraining we use a decontamination tool to ensure minimal contamination with evaluation datasets. We focus our decontamination efforts on the midtraining stage (and the long-context extension, which drew from the same data pools) in light of results suggesting that memorization occurs most strongly near the end of training ( Magar and Schwartz, 2022 ; Bordt et al., 2024 ) .

##### Method and tooling

For decontamination, we search for and remove matches of any split of any benchmark dataset that are part of in our evaluation harness, as for some we increased sample size by evaluating on training splits. We detect and remove contamination between midtraining data and benchmark documents by developing a new decon package 21 21 21 github.com/allenai/decon . Briefly, decon operates in two phases:

1. Detection phase For each midtraining document, decon samples n-grams at a regular stride, checking whether the current n-gram matches known n-gram for any benchmark in the evaluation suite 22 22 22 We decontaminate against all benchmarks in the OLMES package: github.com/allenai/olmes .

2. Cluster expansion phase If a match is found, the matching text is expanded on both sides, counting the number of adjacent ngrams that are also contaminated; if the value is above a specified threshold, the document is deemed contaminated removed.

The two phases approach is key for efficiency: detection phase checks at non-overlapping intervals to speed up processing, while the cluster expansion phase thoroughly checks for matches to compute an accurate contamination score.

We tune the contamination score to balance precision and recall based on numerous qualitative review.

We iteratively refine our decontamination protocol; For example, the first version fails to decontaminate against SQuAD v2 due to a preprocessing issue; DROP is also incorrectly processed due to its short-question-about-a-passage format. We address these issues by evaluating question, answer, and passage components separately—matching primarily on questions, but using answer/passage matches as supporting information for shorter or edited questions. We also improve precision for multiple-choice evals by matching against full answers rather than just A/B/C/D labels. The decon repository includes configuration files that reproduce both the earlier and final approaches. Appendix A.5 provides a detailed overview of decon .

#### 3.5.4 Key findings

Our two-part methodological framework for evaluating midtraining enables us to track closely the quality of our candidate mixes and the behaviors of individual data sources in interaction with others. Here we detail some of the key findings from that process.

##### Candidate mix quality improves over time

Our integration tests allows us to verify progressive improvements in our candidate midtraining mixes over time: Table 6 shows this improvement across a sample of three candidate mixes illustrating the development trajectory. (Since midtraining development operates in tandem with pretraining, we develop mixes on earlier pretrained checkpoints—thus the comparisons here are given to illustrate progress in data curation, and should not be confused with final midtraining numbers.)