Olmo 3 — part 15/19

Cleaned-text code points [218665, 234468). Bibliography omitted here; source.md includes it.

Next, we demonstrate that quality-aware upsampling outperforms naive quality-based filtering. To simulate a data-constrained 4.51T token training run, we compare different data selection strategies in Table 39 . For the filtering baselines, we select the top percentiles from our vigintile quality buckets and match the resulting repetition factor that would occur when training on 100B tokens drawn from a theoretical 4.51T pool. For the upsampling approach, we apply the same methodology but set the target pool size to 100B tokens directly. Our results show that quality-aware upsampling consistently outperforms flat filtering across all repetition factors.

##### Reconciling upsampling and mixing

Finally, we evaluate how to best combine our mixing and upsampling methodologies, which address complementary aspects of data selection. Data mixing determines the distribution across topics, while quality upsampling determines the distribution within a single source. To conceptualize this, imagine the dataset as a two-dimensional matrix of buckets where rows represent WebOrganizer topics and columns represent the quality buckets. Then the mixing strategy can be thought of as imposing row-wise (topic) constraints only. The quality-aware upsampling experiments in the preceding paragraph impose column-wise (quality) constraints only.

We considered several techniques that did not work quite as well as the truncated power-exponential forms described in § A.2.4 . On one hand, the Olmix framework samples data from each topic (row) according only to the natural quality distribution. On the other, quality upsampling samples data from each quality bucket (column) and does not consider reweighting topic distributions. For a theoretical target token yield, each of these strategies prescribes a target token count to be taken from each (topic, quality) bucket. Naive ways to rectify these strategies is to take an arithmetic or geometric mean between the target token counts from each bucket. We also note that the theoretical framework defining upsampling curves above is not necessarily restricted to the concept class of truncated power-exponential families. We could just as easily consider the family of exponential functions like f λ ​ ( x ) = C ​ e λ ⁡ ( x − a ) f_{\lambda}(x)=Ce^{\lambda(x-a)} . Upon considering each of these techniques on small 1B models, we found that the truncated power-exponential family performed the best. Results are contained in Table 40 .

### A.3 Base Model Additional Data Details: Midtraining

This section provides further detail on curation processes for Dolma 3 Dolmino Mix . Additional replication resources, including prompts for synthetic data generation, are available in the Dolma3 GitHub repository.

#### A.3.1 Math Capabilities

Similar to OLMo 2 , we take particular care to curate math-specific mixes of data during the midtraining phase of training. In this section we discuss some of the procedures used to generate, as well as validate, the math-specific data sources. It should be noted, that while there has been a flurry of research on high-quality, open-source, STEM-focused datasets, many of these are synthetic data generated using LLama-models, which carry with them a restrictive Llama license. We produce several reproductions of these with more permissive licensing and urge the community to take care in the licensing of the data they release if they wish to see adoption for research or commercial purposes.

##### TinyMATH

In OLMo 2 , great strides were made in performance on the GSM8K ( Cobbe et al., 2021 ) dataset by generating synthetic math problems seeded from the original GSM training set, and then generating both python code (PoT) and natural language discussions of solutions (MIND). We adopt a similar strategy here, to target the MATH dataset ( Hendrycks et al., 2021c ) . Namely, we adopt the TinyGSM protocol ( Liu et al., 2023a ) and prompts to generate 100 new problems for each existing MATH problem, and then generate pythonic solutions for each of these new problems. Then we apply the MIND rewrite prompt ( Akter et al., 2024 ) , using the two-student and problem-solving variants. This yields the PoT dataset (241M tokens) and the MIND dataset (899M tokens). To assess the potency of these new datasets, we ran annealing runs and evaluated fine-grained math related benchmarks as well as MMLU, to keep an eye on generalization. These results are summarized in TABLE:

##### CraneMath

SwallowMath ( Fujii et al., 2025 ) is a 2.3 Billion token dataset, generated from rewriting FineMath4 + + ( Allal et al., 2025 ) . Unfortunately the data was rewritten using a Llama model, which would require that any model trained on this data would need to have "Llama" in the name, according to the Llama Community License. To provide truly open data, we mirror the generation of this dataset, but use Qwen3 32B Yang et al. (2025a) to rewrite FineMath4 + + using the prompt presented in the SwallowMath paper. This yields a 5.62B token dataset we refer to as CraneMath. Compared to the 9.6B tokens contained in FineMATH4 + + , CraneMath is a distillation into fewer tokens, but not as few as SwallowMath (2.3B) – we posit that this is because using Qwen3 as a rewrite model is slightly "chattier" than Llama.

To evaluate performance of this rewrite procedure, we ran several anneals, starting from a base model that had seen 6T tokens of our pre-training mix, we ran several anneals, always with 50% token from the pretraining mix and 50% tokens from the data-source of interest. In the case where the anneals have different token counts, driving the learning rate linearly down to the same final learning rate. Then we compare the following runs: i) The pre-anneal baseline, ii) FineMath4 + + , but just an incomplete subset; iii) the original SwallowMath dataset; iv) our version, CraneMath; v) two copies of CraneMath; vi) a copy of CraneMath and all their original documents from FineMath4 + + .

##### MegaMatt

OctoThinker ( Wang et al., 2025 ) generated a 70B token data pool dubbed Megamath-Web-Pro-Max, intended to be a rewrite of LLM360’s MegaMath data pool ( Liu et al., 2023c ) , with quality mirroring that of the MegaMath-Web-Pro quality. Again, unfortunately, MegaMath-Web-Pro-Max was rewritten using Llama, and an independent recreation needed to be performed for fully-open usage in training. Since our initial ablations showed that the Megamath-Web-Pro-Max pool wasn’t as high of quality as, say, SwallowMath, we didn’t need a recreation of the full 70B pool. Instead, we generated a recreation of just the documents from Megamath-Web-Pro-Max that occured in CommonCrawl dumps from dump CC-MAIN-2023-23 and later, since more recent data was shown in the OctoThinker paper to be of higher quality. We ultimately generated 3.9B tokens of data, dubbed MegaMatt. To verify the efficacy, we ran ablations on: i) MegaMath-Web, ii) MegaMath-Web-Pro-Max (both to 10B and 25B tokens), and iii) MegaMatt.

##### OMR Rewrites

Inspired by the success of Nvidia’s OpenMathReasoning dataset on the AIO-2 Kaggle competition, we experimented with various rewrites sourced from AoPS forums Moshkov et al. (2025) . See Dolma3 repo for further details.

##### Key Findings and Results

We summarize the annealing results for the math datasets in Table 41 . Each value reflects the change in the evaluation score relative to the pre-anneal baseline, normalized by the number of training tokens. Presenting the results this way highlights several distinct tiers of math-data quality, stratified by the effect-per-token. Notably, these quality tiers anticorrelate with the number of available tokens: the highest-quality sources are also the smallest. While it is true that there are diminishing returns of evaluation scores as more tokens are added, we claim that amongst these high-quality data sources, some higher quality than others.

At the top of the quality-spectrum are the tinyMATH variants. Although each contains less than a billion tokens, they deliver the strongest improvement per token – this is perhaps not surprising as these tokens were specifically crafted to augment the MATH evaluation score. Next in the tier-list of quality are the synthetic rewrites of natural high-quality data: the Crane, SwallowMath and MegaMatt sources which are each rewrites of FineMath4+ and MegaMathWeb-Pro. These provide a markedly weaker lift to the math evaluation metrics than the tinyMATH variants but also have a much larger pool of tokens to draw from. Finally, the largest data sources, including those of naturally occurring data such as FineMath4+ and MegamathWeb, also yield improvements, but their effect-per-token is noticeably smaller than that of the highly curated synthetic data. Finally we note that the effect of math midtraining on MMLU is generally neutral to negative, but is more strongly negative the more targeted the data pool is to Math evals, suggesting “overcooking”, where increased specialization comes at the expense of broader general-purpose performance.

#### A.3.2 Code Capabilities

During pretraining, we relied entirely on stack-edu ( Allal et al., 2025 ) for providing coding data. This data came in the form of naturally-occurring source code from github scraps with limited extra preprocessing. During midtraining, we focused on improving Python and code-completion capabilities. To this end, we incorporated 10B tokens of FIM-transformed data form the same source as the pretraining code mixture. Inspired by improvements in math metrics by incorporating synthetic data, we also created a fully-open replica of SwallowCode ( Fujii et al., 2025 ) , which we call CraneCode.

##### CraneCode

Of the off-the-shelf synthetic code data sources we considered, SwallowCode provided the greatest lift to coding evaluation metrics. Unfortunately, SwallowCode was generated using Llama models and thus had the less-permissive Llama license attached. We created a replica of SwallowCode by starting with just the python files from The Stack v2 Smol ( Lozhkov et al., 2024 ) , and applying the compilation and linting filters just as in SwallowCode. Then we applied a two-stage rewriting process, first to generate code data that is more compliant to the python style guides (SGCR), and then to generate optimized code (SCOR); both using the prompts from the original SwallowCode paper and Qwen/Qwen2.5-Coder-32B-Instruct ( Qwen et al., 2024 ) . To verify the quality of the reproduced dataset, we ran several anneals, where results are displayed in Table 42 .

#### A.3.3 Thinking Capabilities

##### Meta-reasoning

Recent work demonstrates that structured meta reasoning capabilities present during pre-training and mid-training serve as the foundation for successful reinforcement learning in complex reasoning tasks. Gandhi et al. (2025) showed that models exhibiting verification and backtracking behaviors during base training achieved dramatically superior performance trajectories during mathematical reasoning RL. Therefore, we begin by identifying structured reasoning capabilities that are critical for mathematical problem-solving. We select seven core capabilities that are foundational to mathematical and programming expertise: self-awareness ( Toy et al., 2024 ; Callaway et al., 2022 ) , self-evaluation ( Fleming and Daw, 2017 ) , goal management ( Ackerman and Thompson, 2017 ; Griffiths et al., 2019 ) , hierarchical organization ( Haupt, 2018 ) , backward chaining ( Olieslagers et al., 2024 ) , backtracking and conceptual reasoning ( Markovits et al., 2015 ) . We then design specific tasks that systematically target these capabilities, as shown in Table 43 , and 44 . For instance, Math Error Recovery specifically targets self-awareness, verification, and backtracking by requiring models to experience authentic mistakes and demonstrate recovery processes. Strategy Selection focuses purely on meta-cognitive choice processes, while Conversation Generation integrates all capabilities through educational dialogue. For data generation, we start with existing math ( Luo et al., 2025a ; Moshkov et al., 2025 ) and coding ( Li et al., 2023a ; Hendrycks et al., 2021a ; Ahmad et al., 2025 ) problems and their corresponding correct answers. Following Pandalla dataset, 62 62 62 huggingface.co/datasets/pandalla/pandalla-math-dataset-v1.0 we automatically augment each problem with detailed annotations 63 63 63 We provide the problem and the correct answer as inputs to o4-mini with high reasoning, to synthesize the annotations following the Pandalla-math annotation schema. covering ‘problem classification’, ‘difficulty analysis’, ‘solution approaches’, ‘common pitfalls’, and ‘verification methods’. These rich annotations serve as inputs for our capability-targeted tasks. For example, the ‘common pitfalls’ field directly informs math error recovery generation, while steps in ‘solution approach’ provides structure for backward chaining tasks. Using the annotated datasets as foundation, we employ GPT-4.1 and o4-mini to generate training data at scale for each capability-targeted task.

##### Existing thinking traces

The full list of existing thinking traces is as follows:

1. General reasoning mix is a compilation of three existing datasets: GeneralThought-430K 64 64 64 huggingface.co/datasets/RJT1990/GeneralThoughtArchive , OpenThoughts-114k ( Guha et al., 2025b ) , and Open-R1-Math-220k 65 65 65 huggingface.co/datasets/open-r1/OpenR1-Math-220k . The resulting dataset contains questions, thinking traces, and answers for topics spanning math, code, natural sciences, humanities, social sciences, and puzzles.

2. Gemini reasoning traces , introduced by Muennighoff et al. (2025b) , contains thinking traces covering domains of math, astronomy, biology, chemistry, computer science, geography, physics, English, law, logic, and more.

3. OpenThoughts2 reasoning traces from Guha et al. (2025b) contains thinking traces in domains of math, science, code, and puzzles.

4. Llama Nemotron reasoning traces ( Bercovich et al., 2025 ) contains thinking trace data for math, code, general reasoning, and instruction following.

5. QwQ reasoning traces consists of the QwQ subset of the OpenMathReasoning dataset ( Moshkov et al., 2025 ) .

Filtering steps included subselecting for permissively-licensed generations, filtering to remove empty and truncated responses, performing checks of verifiable claims and safety, filtering overt LLM self-references, filtering heavily repeated sentences, paragraphs, and phrases, and remove reasoning traces consisting of greater than 5% Chinese characters.

### A.4 Base Model Additional Evaluation Details

The OlmoBaseEval suite expands on the 11 tasks in the OLMo 2 iteration of OLMES ( OLMo et al., 2024 ; Gu et al., 2024b ) , to include 43 tasks across new families of capabilities. Here, we enumerate details from Section § 3.3 . All task suites are publicly available at github.com/allenai/olmes#olmo-3-eval-suite .

##### Expanding OLMES tasks

We expanded our evaluation to target specific capabilities: new QA tasks focusing on science knowledge (SciQ, QASPER, SciRIFF), medical/lab knowledge (ProtocolQA, DBQA, MedMCQA, MedQA), math tasks (GSM Symbolic, Minerva MATH) and coding tasks (DS 1000, BigCodeBench, Deepseek LeetCode 66 66 66 We use ‘Deepseek LeetCode’ to refer to the 180 LeetCode problems used during development in Guo et al. (2024) , MultiPL-E HumanEval, MultiPL-E MBPP). We use MultiPL-E to evaluate our multilingual code execution, limited to six core programming languages. Additionally, we track fill-in-the-middle (FIM) performance using HumanEval with the three settings from Bavarian et al. (2022) : single-line infilling, multi-line infilling and random span infilling.