Olmo 3 — part 13/19

Cleaned-text code points [186919, 202849). Bibliography omitted here; source.md includes it.

Following our Olmo 3 Think recipe, we train Olmo 3 Instruct on a mixture of general chat, math, and code data. 56 56 56 Preliminary experiments indicated that alternative RL setups—for example, first warming up on math-only data and then switching to a mixed dataset without math—resulted in suboptimal performance. We likewise employ OlmoRL for training, with a maximum response length of 8K tokens for 7B and 16K for 32B 57 57 57 We experiment with both 8K and 16K length training for 7B and 32B; while evaluation scores are minimally impacted by different lengths, we notice undesirable behaviors when qualitatively testing 7B-16K and 32B-8K configurations in an internal demo. . Since our goal for Olmo 3 Instruct is to avoid generating excessively long outputs and preserve general usability, we apply RL on top of two DPO candidates: one that achieved the best average performance, and another with slightly lower performance but better qualitative “vibe test.” We then choose the final RL checkpoint based on final average performance, length analysis, and vibe test. Concretely, we begin by ranking checkpoints by average score; in the case of ties, we place more emphasis on datasets that do not scale with test-time compute (e.g., MATH and AIME performance increase with response length) to avoid biasing our selection towards models with overly long responses. Finally, we apply the vibe test to identify regressions or undesirable behaviors that may fall outside the scope of our evaluation suite.

### 5.5 Key Findings

Below, we summarize our key findings across all 3 stages of Olmo 3 Instruct training:

##### Starting from the Olmo 3 Think SFT is helpful

We find that training Olmo 3 Instruct on top of the Olmo 3 Think SFT both increases model performance on benchmarks, as shown in Table 29 . Importantly, average model response length is minimally affected by this strategy: Olmo 3 Instruct SFT checkpoints produce succinct answers with no remnants of thinking traces.

##### High contrast in preference pairs drives DPO improvements

We observe that a high contrast between completions is critical for achieving improvements during DPO training (Table 32 ). Using LLM-judge pipelines requires carefully thinking about maximizing the delta between chosen and rejected responses. Our initial attempts to modernize the OLMo 2 preference data pipeline by improving the models used to generate responses failed to yield any improvements beyond the OLMo 2 data baseline (Table 32 ). This is likely because the models used for synthetic completions were universally too good: the chosen and rejected responses no longer had meaningful contrast. Extending prior findings that high contrast pairs are critical for performance ( Geng et al., 2025 ; D’Oosterlinck et al., 2025 ) , we introduce interventions to explicitly lower the quality of the rejected response and therefore increase the magnitude of the quality delta in the preference pair. These resulting delta-aware GPT pairs significantly outperform the OLMo 2 preference data.

##### Combining different preference signals improves overall performance

We combine delta-learning heuristic data with GPT-judged preference pairs to get the “best of both worlds.” Empirically, tuning with either delta-learning or GPT-judged pairs yields a different spread of gains; we find that these gains are complementary. Combining both sources of preference signal outperforms using either alone (Table 32 ).

##### The ideal amount of preference data depends on the downstream task

Preference-tuned model performance peaks with different amounts of training for different downstream task domains. We plot preference-tuning performance for example tasks across varying amounts of delta-learning heuristic pairs 58 58 58 Initial experiments with GPT-judged data showed similar trends. in Figure 23 . Further optimization beyond these optimal points hurts downstream performance, consistent with theoretical results showing that early stopping is important for preference tuning ( Azar et al., 2023 ; Geng et al., 2025 ) . Practically, this informs our training approach: we sweep learning rate and dataset size to control the amount of total optimization, and pick the best-performing setting via our development evaluation set.

##### Concise, usable model outputs from preference tuning can boost RL performance

Applying length control during DPO substantially reduces the model’s average generation length, allowing us to trade off some performance for improved conciseness and overall usability. While this reduction in length comes with lower scores on length-sensitive evaluations—particularly math benchmarks such as AIME and MATH—our internal qualitative assessments (“vibe tests”) almost uniformly preferred the shorter, more direct model. We make a conscious decision to prioritize usability.

Crucially, despite the lower benchmark performance at the DPO stage, length control ultimately yields to a more performant model post RL. At 7B scale, we conjecture that this arises from the RL training context window: with a fixed context window (8K), a shorter model may be “more intelligent per token,” allowing it to leverage the available budget more effectively during optimization. Thus, what initially appeared to be a tradeoff between usability and performance ultimately produced improvements in both. Moreover, we found that RL training progresses more reliably when initialized from the length-controlled DPO policy. Across most benchmarks, performance improves more steadily compared to RL runs starting from a higher-scoring but uncorrected DPO checkpoint, which tends to show earlier signs of instability or degradation. This further supports the role of concise preference-tuned models as advantageous starting points for RL.

##### Need for tools

We assess how much of Olmo 3 Instruct ’s performance on LitQA2 and SimpleQA can be attributed to tool use by measuring the delta of the model performance on the benchmarks between answering the questions only from parametric memory (“No tools” setting) and doing so using tools. Table 31 shows these deltas in comparison to those from three Qwen models. All models benefit significantly from tool use on SimpleQA. However, Qwen models, unlike Olmo 3 Instruct 7B, mostly seem to rely on parametric knowledge for LitQA2, with two of the models even losing performance when provided with tools.

LitQA2 SimpleQA No tools ASC Δ \Delta No tools SBT Δ \Delta Olmo 3 Instruct 7B 24.4 38.2 13.8 3.3 79.2 75.9 Qwen 3 8B (w/o reasoning) 34.7 39.6 4.9 2.0 79.0 77.0 Qwen 3 VL 8B Instruct 34.7 30.7 -4.0 9.3 90.3 81.0 Qwen 2.5 7B 36.0 29.8 -6.2 3.3 78.0 74.7

## 6 Olmo 3 RL-Zero

RL has become a key part of recent LLM pipelines in part due to prominent open models such Deepseek R1-Zero ( Guo et al., 2025 ) , which notably leverages RL training on top of a base model to bootstrap complex reasoning behavior ( Marjanović et al., 2025 ) , and due to the rapid adoption of closed reasoning models such as OpenAI’s o1-series and Gemini with Thinking. This has made RLVR finetuning from a base model the standard large-scale benchmark for RL algorithms ( Liu et al., 2025a ; Yu et al., 2025 ; Luo et al., 2025b ) . To date, leading open RLVR benchmarks and algorithms train on top of open-weights models that do not reveal their pretraining or midtraining data ( Chu et al., 2025 ; Yang et al., 2025a ) . This limits the ability for the community to study the role of pretraining data on RLVR performance. It can lead to a myriad of issues with benchmark evaluations being contaminated, e.g., midtraining data containing data from the evaluation set, which makes spurious rewards as effective as true rewards ( Shao et al., 2025b ; Wu et al., 2025c ) , or improvements from fixing prompt templates outweighing the improvements from RL ( Liu et al., 2025b ) .

We therefore release a fully-open dataset Dolci RL-Zero , an algorithmic RL zero setup for Olmo 3 , and open-source OlmoRL code to enable clear benchmarking for the ecosystem. We perform RLVR from Olmo 3 Base over five benchmarking domains to create the Olmo 3 RL-Zero family: math, code, precise instruction following (IF), general chat, and a mix of all listed sub-domains. In all cases, we further decontaminate Dolci RL-Zero from pretraining and midtraining data to guarantee our setup carefully studies the effect of RLVR without data leakage confounding our conclusions.

### 6.1 Reinforcement Learning From Base with Dolci RL-Zero

##### Data

We create Dolci RL-Zero , an effective RL-Zero training dataset. For math, we aggressively filter DAPO Math ( Yu et al., 2025 ) , Klear-Reasoner Math ( Su et al., 2025c ) , Open-Reasoner-Zero (Orz) ( Hu et al., 2025 ) , and Omega ( Sun et al., 2025 ) . We deduplicate DAPO and remove all non-English examples. As Klear-Reasoner, Orz, and Omega are much larger datasets, we further group questions via semantic clustering across Klear-Reasoner, Orz, and Omega, and select one representative question per cluster, in addition to including DAPO. We further decontaminate against both pretraining and evaluation data following subsubsection 4.2.1 and perform offline filtering, removing prompts fully solved in 8 out of 8 sample completions by the final base model. This results in a dataset of 13.3K math prompts. Data for code, instruction-following, and general chat are subsampled from Dolci Think RL (Section § 4.4.2 ).

##### Prompt and eval template

Confirming the findings of Liu et al. (2025b) , we find that “simple” prompt templates greatly outperform standard post-trained templates (e.g., <think></think> ) when training from a purely midtrained model, as Dolma 3 Dolmino Mix excluded most special formatting. We develop a simple custom prompt for each domain, using the zero-shot pass@k performance as our metric. We end up with a prompt similar to Yu et al. (2025) , shown in Figure 37 . We furthermore “clean” all our evaluation prompts to remove special formatting (i.e., \boxed{} ) to make evaluation prompts more similar to our training prompts.

##### RL algorithm

We follow Section § 4.4.1 in all RL details except (i) we train with a response length of 16K tokens to better accommodate long chain-of-thought reasoning in the math and code domains and (ii) we evaluate with a response length of 32K tokens and temperature 1.0 to encourage diversity as we report pass@k. See Table 49 for hyperparameter details.

### 6.2 Key Findings

##### Olmo 3 RL-Zero can strongly improve on reasoning

As shown in Figure 24 , our base model can greatly improve on training reward across the different domains when leveraging RL on our datasets. To demonstrate out-of-domain improvements, we evaluate our math run on the decontaminated evals AIME 2024 and 2025. We find that Olmo 3 Base drastically improves in the first couple hundred steps of training and then improves steadily but slowly. We also see a decent improvement in pass@32 results, demonstrating that our run maintains diversity and RLVR pushes the model beyond its initial capabilities. Our initial scores and final scores with the 7B model are, notably, close to DAPO ( Yu et al., 2025 ) which leverages the larger Qwen 2.5 32B and trains for an order of magnitude more steps, see Figure 38 in Appendix A.6.4 . This demonstrates how Olmo 3 RL-Zero can be a more efficient alternative to existing RLVR experiments.

##### Olmo 3 RL-Zero mix can benchmark challenges in multi-objective RL

Most studies have focused exclusively on RLHF ( Stiennon et al., 2020 ) or single-domain RLVR ( Yu et al., 2025 ; Luo et al., 2025a ) . Our mix of math, code, instruction-following, and general chat is a more challenging RLVR benchmark for models. Figure 24 demonstrates that our general run has improved performance across different domains, but each domain is under-optimized compared to the single-domain setup. Future work can leverage this setup to investigate the interactions between domains in multi-objective RLVR.

##### Olmo 3 RL-Zero can benchmark reasoning data mixes in midtraining

Midtraining and Olmo 3 RL-Zero offer a chance to ablate specific data sources, unlike the large-scale effort behind Olmo 3 Think . We leverage RL-Zero to evaluate midtraining data mixes for their ability to develop downstream reasoning with RL. For example, we compare two early models in Figure 25 . As evidenced by the stagnant response length, the model with insufficient reasoning data does not leverage backtracking, answer verification, and other cognitive skills ( Gandhi et al., 2025 ) . Olmo 3 RL-Zero can therefore serve as a testbed for downstream performance of alternative midtraining approaches and improvements over Dolma 3 Dolmino Mix .

##### Active sampling stabilizes training

Olmo 3 RL-Zero also offers a simpler testbed for ablating RL algorithm and infrastructure decisions. We ablate active sampling, our novel method for continuously resampling prompts after filtering for non-zero advantage (see Section § 4.4.3 for details). Running on our math domain, Figure 26 shows that active sampling does indeed maintain a consistently full batch of completions with non-zero advantage. These consistent batch sizes have a stabilizing effect on training, and we see greatly reduced loss variance.

##### Eval decontamination is verified via spurious rewards

Recent RLVR benchmarks have shown substantial improvements from training with spurious rewards that are not correlated with model utility. This can suggest that the RLVR task may have been contaminated , i.e., the model was exposed to evaluation data during pretraining or midtraining. RLVR with a spurious reward can elicit this memorized knowledge, differentiating it from genuine learning of reasoning capabilities ( Shao et al., 2025b ) . To verify that Olmo 3 RL-Zero evaluation is not contaminated, we conduct a negative control experiment by training Olmo 3 Base with spurious rewards. Specifically, we train on Dolci RL-Zero , but instead of rewarding correct answers, we assign random binary rewards to model generations independent of response quality following the protocol in Shao et al. (2025b) . If our pretraining or midtraining data contained significant overlap with our evaluation sets, we would expect spurious reward training to elicit these memorized solutions and improve benchmark performance.

As shown in Figure 27 , training with random rewards does not improve performance on any of our benchmark evaluations. Performance either remains flat with random fluctuations or degrades, which is consistent with the model learning arbitrary patterns unrelated to the task. This negative result is evidence that our data decontamination successfully removed overlaps between our base-model pipeline and RLVR evaluation data.

## Appendix A Appendix

## Author Contributions

A successful team project like Olmo 3 would not be possible without the contributions of many teammates. We indicate each authors’ main contributing role(s) in Olmo 3 , while recognizing that project impact was driven by fluid contributions across formal team boundaries. Authors are listed in alphabetical order:

• For model architecture, infrastructure, and training methodology: Akshita Bhagia, Aman Rangapur, Amanda Bertsch, David Heineman, Dirk Groeneveld, Dustin Schwenk, Kyle Lo, Luca Soldaini, Mayee Chen, Pete Walsh, Shane Arora, Tyler Murray, Tyler Romero, Will Merrill

• For post-training infrastructure and training methodology: Costa Huang, Faeze Brahman, Finbarr Timbers, Hamish Ivison, Jacob Morrison, Michael Noukhovitch, Nathan Lambert, Pradeep Dasigi, Saurabh Shah, Scott Geng, Shannon Zejiang Shen, Shashank Gupta, Teng Xiao, Tyler Romero, Valentina Pyatkin, Victoria Graf

• For base model data acquisition: Chloe Anastasiades, David Graham, Dustin Schwenk, Jake Poznanski, Jaron Lochner, Kyle Lo, Luca Soldaini, Matt Jordan, Robert Berry, Tyler Murray