Olmo 3 — part 11/19

Cleaned-text code points [155608, 171544). Bibliography omitted here; source.md includes it.

We employ continuous batching to constantly enqueue new generations as each one finishes to remove the compute waste for long generations (see Figure 17 ). This is in contrast to static batching , in which a batch of prompts is split over N N actors, and each actor generates the entire batch, 42 42 42 Calling llm.generate in vLLM. returns the generated responses to the learner, and then receives a new batch of data. Static batching is inefficient, as when one generation finishes that “slot” of the batch will remain empty until we get a new batch. The exact wasted compute can be calculated as the maximum sequence length minus the average sequence length divided by the maximum sequence length. With Olmo 3 , at a 32K generation length, we see a mean generation length of 14628 and a maximum of 32K, which means that up to 54% of our compute would have been wasted with static batching. See Figure 17 for an illustrated example.

##### Active sampling

To compensate for filtered instances, our fully asynchronous framework enables continuously pulling completions from the actor and resampling prompts into the queue. We actively sample and filter until we reach our desired batch size of non-zero-gradient completions. Previously, Yu et al. (2025) dynamic sampling would oversample and generate three times the number of prompts used in each training batch. This was to reasonably guarantee that the batch had enough completions with non-zero standard deviation. In contrast, our active sampling more efficiently uses the infrastructure. As demonstrated in section 6 , we find this significantly stabilizes training and prevents batch size from reducing over the course of training (a common issue with vanilla GRPO).

##### Inflight updates

A common goal of RL training for LLMs is to minimize the degree of difference between the actor policy and the learner policy, i.e., to minimize being off-policy ( Van Hasselt et al., 2018 ) . This can be achieved by synchronizing the weights after every training step as follows: each actor finishes all of their ongoing generations, dumps the KV cache, and updates its copy of the weights. However, this causes GPUs to be idle and hurts training efficiency. Instead, we follow Piché et al. (2025) to immediately update the weights without pausing the engine, relying on the generation framework to be thread-safe, and continue generating, without invalidating the KV cache . This enables a significant increase in throughput: up to 4x faster with the same resources, without hurting accuracy.

##### Better threading and engineering

These changes are primarily around handling the weight synchronization after each training step to make actors more efficient. Our new setup decouples the actors, allowing each one to start and stop by itself, without waiting for the rest of the actors to finish their syncs as well. Similarly, we make a large number of optimizations that were not machine learning specific, and were centered around efficiently using the CPU . For example, our initial implementation of continuous batching, for instance, was slower than static batching before adding a prefetch thread to our actors that constantly refilled the inference queue to see a throughput improvement.

Our final RL run ended up mixing carefully-filtered data from all domains roughly equally and running on top of the DPO checkpoint.

### 4.5 Key Findings

##### DPO yields gains where SFT on the same data cannot

Continued supervised finetuning directly on the chosen responses from Dolci Think DPO outright hurts the initial SFT model ( Table 21 ), dropping all evaluation tasks. We conjecture that this is because the chosen responses (generated by Qwen3 32B Thinking) are weaker relative to data the model has already seen in Dolci Think SFT , and hence, they are no longer useful targets for imitation. However, by pairing these chosen responses with rejected responses generated by a weaker model, we construct a useful contrast, enabling preference tuning to drive strong gains beyond the initial SFT model ( Table 21 ). Promisingly, these gains are not merely converting pass@k into pass@1 but rather expanding the reasoning frontier of the model (e.g., improved pass@k on AIME evaluations; Figure 20 ). These findings highlight contrastive learning with preference tuning as a useful stage for improving capabilities even when imitation is saturated.

##### DPO and SFT both benefit from RL, but DPO remains a better starting point

Table 22 shows that running our final RL mix on the DPO model consistently yields better performance than running it on the SFT model. We find three primary differences, highlighted in Figure 19 : for evaluations that RL does not improve, the DPO model often performs better and maintains its advantage during RL training (e.g., AlpacaEval). For evaluations explicitly targeted by RL (e.g., Omega), both the DPO and SFT models achieve similar end performance. For evaluations targeted by RL but hard to improve further from DPO (e.g., AIME 2025), the SFT model improves to get close to DPO performance. In no situation does the SFT model improve over the DPO model after RL, and as such we opt to focus on applying RL over our DPO model. Curiously, we find that the SFT model performs similarly when trained either with the data offline-filtered using the SFT or DPO model, suggesting that the additional samples filtered out (i.e., solved) by the DPO model do not provide additional signal for improving the SFT model. Further investigating this, we find that while the DPO model does display lower entropy, it in fact has higher pass@K performance on AIME evaluations, as shown in Figure 20 . This suggests that the DPO model remains a strong starting point for RL relative to the SFT model, as prior work ( Yue et al., 2025 ; Shao et al., 2024 ) suggests RLVR, under certain conditions, helps convert pass@K improvements into pass@1 gains.

##### Rewards steadily increase across all domains during RL

Figure 18 plots per-verifier reward curves along with average output length. We find that reward steadily increases across all domains, albeit at differing rates (with instruction-following data increasing most steadily, and code reward increasing most slowly). We plot more RL curves in the appendix ( Figure 41 ). Interestingly, we find that sequence lengths first slightly dip and then slowly increase over time. This is likely due to the reasoning SFT and DPO already training the model to produce long reasoning traces of up to the maximum response length of 32K tokens.

##### Mixing RL data from varied domains can prevent over-optimization

Figure 20 (left) demonstrates that training on specific domains can lead to over-optimization, in which performance on evaluations outside that domain drops, while training on a mix yields steady improvements across different domains. For example, we observe a trade-off when performing OlmoRL on IFEval alone, wherein higher IFEval scores correlate with lower AlpacaEval scores. However, when we perform our final mixed training, we are able to maintain high AlpacaEval scores without compromising IFEval performance, as the LM-judge reward ensures that the model continues to produce well-formed chat responses.

##### Mixing data yields lower train reward, but not lower downstream performance

While Figure 20 demonstrates that our final mixture run achieves downstream performance similar to or greater than RL training runs on single domains, we find that we observe lower train reward across each domain when training on mixed data as opposed to single-domain data, as seen in Figure 21 . This suggests that mixing data may in fact reduce the model’s tendency to over-optimize during training , preventing some degree of reward-hacking and thus generalizing better to downstream evaluations. This may explain why RL training on broader data mixtures can outperform domain-specific mixtures ( Cheng et al., 2025 ) .

##### Continuous batching and inflight updates are crucial to training speed

Using a reasoner SFT or DPO as a starting point stresses RL training to its limits, as the model starts with extremely long average generation lengths. Table 23 demonstrates how using continuous batching and inflight updates is crucial to training speed, allowing us to achieve two times faster training on half as many GPUs, making experimentation and long RL runs more tractable. 43 43 43 While an initial checkpoint took 14 straight days of training across 9 nodes to achieve 1 epoch, with continuous batching and inflight updates, we could achieve 1 epoch on 5 nodes in 7 days. To carefully benchmark this, we ablate the changes to our RL infrastructure between OLMo 2 and Olmo 3 . See Table 23 . For each ablation, we ran a benchmark experiment for 2 hours using 2 8x A100 nodes. One node was used for training, and one for inference. Since inference is our bottleneck, we report Model FLOPs Utilization (MFU) and Model Bandwidth Utilization (MBU) based solely on the single node used for inference. A typical full-scale experiment would use many more nodes for inference, typically with a 8:1 ratio (or more) of inference nodes to training nodes. The benchmark experiment generates a batch of 128 completions for each training step, using 64 prompts, each sampled twice, with a maximum output length of 32000, and a maximum input length of 2048, leading to a context length of 2048. 44 44 44 Script can be found in the github.com/allenai/open-instruct , at scripts/benchmarking/olmo3_infra.sh .

##### OlmoRL shows significant improvement in precise instruction following

The precise instruction-following performance increases across post-training stages, with the final RL training stage leading to the biggest improvements in Olmo 3 ’s precise instruction-following abilities, as shown in Table 24 , for both the development (IFEval) and the unseen (IFBench) evaluations.

## 5 Olmo 3 Instruct

Recent studies suggest that real-world language model use predominantly centers on general tasks such as advice-seeking and information recall ( Chatterji et al., 2025 ) that may not require extensive reasoning. Everyday chat settings often do not require the inference-time scaling of Olmo 3 Think . Hence, we develop Olmo 3 Instruct , a non-reasoning model designed with these real use cases in mind. Olmo 3 Instruct quickly and helpfully respond to common user queries.

This different model type demands different data to support it. We focus on improving the interactivity of the models by introducing multi-turn DPO data and promoting concise responses in our delta-learning preference-tuning pipeline. Additionally, Olmo 3 Instruct is trained for function-calling, for which we release new SFT datasets. Together, our recipe yields Olmo 3 Instruct models that effectively leverage tools and efficiently respond to user queries.

### 5.1 Main Results for Olmo 3 Instruct

Olmo 3.1 32B Instruct Baselines SFT DPO Final Instruct 3.1 Apertus 70B Qwen 3 32B (No Thinking) Qwen 3 VL 32B Instruct Qwen 2.5 32B Gemma 3 27B Gemma 2 27B OLMo 2 32B Math MATH 74.4 86.6 93.4 36.2 84.3 95.1 80.2 87.4 51.5 49.2 AIME 2024 12.7 35.2 67.8 0.31 27.9 75.4 15.7 28.9 4.7 4.6 AIME 2025 8.2 23.3 57.9 0.1 21.3 64.2 13.4 22.9 0.9 0.9 OMEGA 15.5 33.3 42.2 5.6 23.4 44.0 19.2 24.0 9.1 9.8 Reasoning BigBenchHard 69.0 82.1 84.0 57.0 80.4 89.0 80.9 82.4 66.0 65.6 ZebraLogic 30.6 51.1 61.7 9.0 28.4 86.7 24.1 24.8 17.2 13.3 AGI Eval English 71.7 79.4 79.5 61.6 82.4 89.4 78.9 76.9 70.9 68.4 Coding HumanEvalPlus 80.8 85.7 86.7 42.9 83.9 89.3 82.6 79.2 67.5 44.4 MBPP+ 61.5 63.6 65.1 45.8 67.9 69.0 66.6 65.7 61.2 49.0 LiveCodeBench v3 35.4 49.6 54.7 9.7 57.5 70.2 49.9 39.0 28.7 10.6 IF IFEval 87.7 87.3 88.8 70.4 87.5 88.1 81.9 85.4 62.1 85.8 IFBench 29.7 36.3 39.7 26.0 31.3 37.2 36.7 31.3 27.8 36.4 Knowledge & QA MMLU 79.0 81.9 80.9 70.2 85.8 88.7 84.6 74.6 76.1 77.1 PopQA 23.7 28.5 25.0 33.5 25.9 25.7 28.0 30.2 30.4 37.2 GPQA 41.3 47.9 48.6 27.9 54.4 61.4 44.6 45.0 39.9 36.4 Chat AlpacaEval 2 LC 42.2 69.7 59.8 19.9 67.9 84.3 81.9 65.5 39.8 38.0 Tool Use SimpleQA 82.3 85.3 84.7 - 86.7 91.5 90 - - - LitQA2 47.6 53.3 55.6 - 46.7 32 26.2 - - - BFCL 57 58.6 58.8 - 63.1 66.3 62.8 - - - Safety 92.1 88.9 89.5 77.1 81.6 85.8 82.2 68.8 74.4 84.2

Olmo 3 7B Instruct Baselines SFT DPO Final Instruct Qwen 3 8B Qwen 3 VL 8B Inst Qwen 2.5 7B OLMo 2 7B Inst Apertus 8B Inst Granite 3.3 8B Inst Math MATH 65.1 79.6 87.3 82.3 91.6 71.0 30.1 21.9 67.3 AIME 2024 6.7 23.5 44.3 26.2 55.1 11.3 1.3 0.5 7.3 AIME 2025 7.2 20.4 32.5 21.7 43.3 6.3 0.4 0.2 6.3 OMEGA 14.4 22.8 28.9 20.5 32.3 13.7 5.2 5.0 10.7 Reasoning BigBenchHard 51.0 69.3 71.2 73.7 85.6 68.8 43.8 42.2 61.2 ZebraLogic 18.0 28.4 32.9 25.4 64.3 10.7 5.3 5.3 17.6 AGI Eval English 59.2 64.0 64.4 76.0 84.5 69.8 56.1 50.8 64.0 Coding HumanEvalPlus 69.8 72.9 77.2 79.8 82.9 74.9 25.8 34.4 64.0 MBPP+ 56.5 55.9 60.2 64.4 66.3 62.6 40.7 42.1 54.0 LiveCodeBench v3 20.0 18.8 29.5 53.2 55.9 34.5 7.2 7.8 11.5 IF IFEval 81.7 82.0 85.6 86.3 87.8 73.4 72.2 71.4 77.5 IFBench 27.4 29.3 32.3 29.3 34.0 28.4 26.7 22.1 22.3 Knowledge & QA MMLU 67.1 69.1 69.1 80.4 83.6 77.2 61.6 62.7 63.5 PopQA 16.5 20.7 14.1 20.4 26.5 21.5 25.5 25.5 28.9 GPQA 30.0 37.9 40.4 44.6 51.1 35.6 31.3 28.8 33.0 Chat AlpacaEval 2 LC 21.8 43.3 40.9 49.8 73.5 23.0 18.3 8.1 28.6 Tool Use SimpleQA 74.2 79.8 79.3 79.0 90.3 78.0 - - - LitQA2 38.0 43.3 38.2 39.6 30.7 29.8 - - - BFCL 48.9 49.6 49.8 60.2 66.2 55.8 - - - Safety 89.5 89.9 87.6 78.4 77.7 73.4 91.1 71.1 74.3

Table 26 and Table 25 demonstrates the results of Olmo 3 Instruct 7B and 32B, respectively, on our evaluation suite 45 45 45 We omit reporting of Essential AI’s Rnj-1 Instruct ( Vaswani, 2025 ) due to discrepancies between our observed and their reported numbers. Qualitatively, Rnj-1 behaves like a code specialized model (generates code even for IFEval and Safety chat tasks). Our evaluation framework is meant for general instruct models without code execution for chat tasks. This yields lower scores for Rnj-1 than they report (e.g., 16.1 versus 43.3 on AIME 25, 64.8 versus 75.7 on MBPP+, 79.3 versus 83.5 on HumanEval+) even when we use their recommended general system prompt for turning off code-producing behavior. Thus, we omit it from comparison as we do other specialized models (eg Qwen Coder). . In addition to the evaluations used for Olmo 3 Think (Section 4.1 ), we add benchmarks for function-calling. 46 46 46 For missing function-calling evaluations: OLMo 2 Instruct and Gemma 2 and 3 don’t support this. Apertus and Granite aren’t supported by BFCL and we had some difficulties getting the other tasks running. We will update the paper with scores as open git requests are resolved. Olmo 3 Instruct 7B outperforms Qwen 2.5-7B Instruct, OLMo 2 Instruct 7B, and Apertus 8B Instruct. Similarly, Olmo 3.1 Instruct 32B outperforms most open models at similar scale, including Qwen 2.5 32B, Qwen 3 32B (No Thinking), Gemma 3 27B, and Apertus 70B. Notably, Olmo 3.1 Instruct 32B achieves 39.7 on IFBench outperforming Qwen 3 and Qwen 3 VL at 32B scale. In addition, Olmo 3.1 Instruct 32B achieves 57.9 on AIME 2025, surpassing Qwen 3 32B (No Thinking) by 36.6 points, and closing the gap to Qwen 3 VL 32B-Instruct.

### 5.2 Supervised Finetuning with Dolci Instruct SFT

We construct Dolci Instruct SFT by building upon our OLMo 2 Instruct mixture, making significant improvements to advance general chat, reasoning, and function-calling capabilities.

#### 5.2.1 Function-calling Training Data

Our goals for curating tool-use training data for Olmo 3 Instruct are to provide the model a strong foundation in basic function calling and to expose the model to trajectories demonstrating the effective use of real environments (i.e., MCP servers) to perform tasks. Accordingly, we collect two kinds of trajectories synthesized using LLMs, described below.

##### Trajectories with real interactions