Olmo 3 — part 12/19

Cleaned-text code points [171544, 186919). Bibliography omitted here; source.md includes it.

We collect trajectories demonstrating agents’ use of MCP servers to answer queries. All trajectories have a single user turn and multiple agent–environment interactions. We focus on the following domains: • Science QA dataset contains two broad classes of queries requiring retrieval and reasoning over scholarly content: 1) paper content-based queries, which focus on information present in the abstract or full text of papers and 2) citation graph-based queries, which are about metadata such as authors, venues, and citations. Trajectories associated with the queries are obtained using an agent based on GPT-4.1-mini equipped with the ASTA Scientific Corpus (ASC) MCP server 47 47 47 allenai.org/asta/resources/mcp , which provides structured access to metadata and paper content on Semantic Scholar 48 48 48 www.semanticscholar.org . Additional details about these datasets are provided in Appendix A.7.2 .

• Web search QA dataset is adapted from DR Tulu ( Shao et al., 2025a ) . It consists of a multi-stage pipeline that combines benchmark-derived and real-world queries. Queries are drawn from open-access benchmarks: HotpotQA ( Yang et al., 2018 ) , TaskCraft ( Shi et al., 2025 ) , and WebWalkerQA (silver) ( Wu et al., 2025a ) , as well as from consented, publicly released user prompts from SearchArena ( Miroyan et al., 2025 ) and OpenScholar ( Asai et al., 2024 ) . We filter the set of queries using GPT-5 to keep only those that both require search and have long-form, verifiable responses. The trajectories for these queries are obtained from a GPT-5 agent equipped with the Serper API 49 49 49 serper.dev , which provides access to a Google search tool and a tool for fetching webpages given their URLs. Additional details about query filtering and trajectory generation can be found in Appendix A.7.2 .

##### Trajectories with simulated interactions

While training on trajectories with executable environments is expected to teach the model to effectively deal with real environment outputs and handle unexpected errors, it is difficult to curate such trajectories at scale, thus potentially limiting the model’s generalization to unseen tools at inference time. To fill this gap, we also create a dataset of synthetic trajectories with LLM-simulated environments which are much easier to scale. We call this dataset SimFC . We start with a large pool of tool sets or APIs from existing datasets (e.g., xLAM ( Liu et al., 2024c ) , ToolACE ( Liu et al., 2024b ) ), and from publicly available MCP servers, and prompted LLMs (GPT-4o, GPT-4.1, and GPT-5) to generate entire trajectories including simulated user queries, environment responses, and assistant messages. We design prompts to ensure the dataset contains a variety of interaction patterns including multi-turn, multi-step, and refusals due to inadequate information or tools. Additional details about this dataset and illustrative prompts used for generation can be found in Appendix A.7.2 , Figure 42 , and Figure 43 .

##### Balancing function diversity with interaction complexity

As illustrated by the statistics in Table 27 , the two types of trajectories have key differences. SimFC has a large number of trajectories with diverse sets of functions. We find that synthesizing trajectories with multiple user turns (multi-turn trajectories) is relatively easier than those with multiple assistant-environment interactions per user request (multi-step trajectories). However, the latter class usually corresponds to more complex tasks. On the other hand, the datasets with real interactions, while smaller in size, are naturally more complex in terms of multi-step interactions.

##### Unified data format

Across all tool-use data, we adopt consistent tool definition and tool-calling formats. We find that unifying format to be crucial for stable and high-quality tool-use behavior. Particularly, we use the OpenAPI specification 50 50 50 swagger.io/specification/ for all tool definitions and represent all function calls as pythonic code blocks. We provide tool specifications in the system prompt, encapsulate tool calls with XML tags within the assistant role, and present environment outputs to the model within a special environment role. We also extend the tokenizer’s vocabulary with dedicated special tokens corresponding to these tags. Unlike Olmo 3 Think , preliminary suggest this approach to be more effective for tool-use training than encoding <functions> , </functions> , <function_calls> , and <function_calls> as regular text.

##### Evaluating function calling

We evaluate the function calling capabilities of Olmo 3 Instruct in terms of intrinsic function calling and extrinsic task completion accuracies using different benchmarks. We use the Berkeley Function Calling Leaderboard (BFCLv3) ( Patil et al., 2025 ) to evaluate intrinsic function calling accuracy. This benchmark focuses on models’ ability to choose the relevant functions and the right values for their arguments to accomplish a given task in settings that require one or more interactions with simulated users and environments. We evaluate task completion accuracy of Olmo 3 Instruct in comparison with similar models when they are deployed as agents with access to tools served via Model Context Protocol (MCP) servers. Particularly, we use the Asta Scientific Corpus (ASC) tool ( Bragg et al., 2025 ) that serves eight functions for accessing scientific literature, and the Serper API which provides Google search tool and web browsing functionalities. To evaluate models’ usage of the ASC tools, following Bragg et al. (2025) , we use a subset of 75 questions from LitQA2 ( Skarlinski et al., 2024 ) for which the associated papers can be found in ASC’s index. We evaluate the models’ usage of search and browsing tools using a subset of SimpleQA 51 51 51 huggingface.co/datasets/akariasai/sampled_simpleqa ( Wei et al., 2024 ) .

We use the official Gorilla repository 52 52 52 github.com/ShishirPatil/gorilla for BFCLv3 evaluations. For LitQA2 and SimpleQA, we implement a basic function-calling agent using OpenAI’s Agent SDK. This agent uses the tools provided by the relevant MCP server 53 53 53 We the same setup introduced by Shao et al. (2025a) for DR Tulu. , and interacts with the environment by iteratively making function calls and processing the outputs of executing them to solve the given tasks. For LitQA2 and SimpleQA, we also measure model performance when deployed in a No-Tools setting, in which we provide no tools to the agents and they are expected to solve the tasks entirely from the models’ parametric knowledge. We use a zero-shot evaluation for all these benchmarks. We sample from models at temperature 0 and, for LitQA2 and SimpleQA, allow the agents at most 10 turns to finish each task. We run each evaluation three times and report the average accuracy. We release our code 54 54 54 github.com/allenai/mcp-tool-eval for running our MCP-based tool-use evaluations.

#### 5.2.2 Curating Dolci Instruct SFT

##### Step 1. Sourcing Prompts and Completions

Our prompt collection includes all our new function-calling data (Section § 5.2.1 ), new prompts for instruction following (see Section § 4.2.1 ) and science, and more chat prompts from WildChat ( Zhao et al., 2024a ) . For examples that originally contained reasoning traces (such as the OpenThoughts3 science subset described in Section § 4.2.1 ), we remove the reasoning traces and special tokens. We also update completions from older models such as GPT-3.5 and GPT-4 with completions from GPT-4.1. We show a summary of our instruct SFT mix in Table 30 .

##### Step 2: Filtering & Mixing

We follow the same filtering and mixing procedure detailed in Section 4.2.1 . For Olmo 3 Instruct , our base mix is 100K examples from an updated intermediate mix based on the OLMo 2 SFT mix. We show results of our data-mixing experiments on OLMo 2 in Table 28 .

##### Starting from Olmo 3 Think SFT

We train the SFT stage of Olmo 3 Instruct starting from the Olmo 3 Think SFT model as shown in Figure 2 to give it a “warm-start.” We found that this significantly improves the performance of the Instruct model, as shown by the results in Table 29 .

### 5.3 Preference Tuning with Dolci Instruct DPO

We create Dolci Instruct DPO by extending the strong base of our delta-learning heuristic preferences (Section § 4.3 ) with further curated preference signals to enhance our model’s behavior in general use settings. We enrich our heuristic data with contrastive pairs from an improved GPT-judge pipeline for general alignment. Additionally, user interaction with LMs commonly requires multi-turn conversational capabilities, so we introduce synthetic multi-turn conversations to our preference data. We also observe that preference-data pipelines often promote overly verbose responses; we introduce counteracting interventions to promote brevity in model responses by mitigating length bias in the preference data.

#### 5.3.1 Preference Signals

Dolci Instruct DPO is constructed from a composite of several preference signals to promote model capabilities and general usability:

##### Delta-learning heuristic pairs

Similar to Dolci Think DPO , we construct heuristic contrastive pairs by generating chosen responses with a large model (Qwen3 32B) and rejected responses with a small model (Qwen3 0.6B) following Geng et al. (2025) . Note that we turn off thinking mode, as we do not need internal thinking traces.

##### Delta-aware GPT-judged pairs

We additionally generate GPT-judged preference pairs to add a further source of preference signal. Our initial attempts to modernize the UltraFeedback pipeline from OLMo 2 and Tülu 3 by improving the quality of the LLM judge (GPT-4o → \rightarrow GPT-4.1) and updating our data-generator model pool do not yield gains and even hurt model performance relative to the OLMo 2 preference dataset baseline. We speculate that this failure is due to the fact that the majority of our data generators are high-quality, very capable models; hence on average there was minimal meaningful contrast between the resulting chosen and rejected pairs. To mitigate this, we explicitly introduce delta-aware interventions designed to lower the quality of the rejected response. We 1) ensure that responses from weaker models are always present in the response set judged for each prompt, and 2) select the worst response as the rejected completion to maximize the resulting delta. We find these “delta-maximizing” interventions to be critical for the quality of preference pair data; see our findings in Section § 5.5 for details.

##### Multi-turn preferences

To ensure Olmo 3 ’s usability in realistic multi-turn conversations, we further add a multi-turn preference dataset with prompts synthetically extended from the Tülu 3-DPO dataset. Preference pairs differed in only the last turn of the conversation to avoid ambiguity in quality ranking between turns of the same conversation. Synthetic conversations are generated with two methods: 1) self-talk extending the original prompt into a multi-turn conversation with LLM-generated follow-up requests and 2) synthetic-context created by generating related, independent questions or paraphrases of the initial prompt to use as previous user turns with associated completions. The combination of these generation methods ensures diversity in generated conversations. Final turns are generated with the delta-learning heuristic ( Geng et al., 2025 ) ; chosen/rejected completion pairs are generated by either GPT-4o and GPT-3.5 or Qwen 3 32B and Qwen 3 0.6B (both no-thinking) respectively.

##### Controlling length bias

Preference data often has a length bias: the chosen responses are significantly longer than the rejected responses. This comes from sourcing synthetic response pairs where historically more information has been treated as more helpful by both LLM judges and preference heuristics. Namely, LLM judges such as the GPT judge in our pipeline tend to prefer longer responses. Similarly, we empirically observe that preference pairs made with the delta-learning heuristic also exhibit length bias; larger models generate longer responses ( Figure 23 ). Thus, models often learn this length bias in addition to the intended useful quality signal during preference tuning, after which its generation length per prompt increases significantly. While this increased length is empirically useful for reasoning tasks, excessive verbosity can be undesirable for common real-use settings (see an example in Figure 22 ). We seek to strike a balance by filtering the chat and multi-turn subsets of our preference data to limit the length difference between the chosen and rejected responses to 100 tokens.

#### 5.3.2 Prompt Mixing

Our prompt pool for GPT-judged and delta-learning heuristic pairs (see Table 30 ) is derived from the Dolci Instruct SFT dataset supplemented with the DaringAnteater and UltraFeedback subsets from the OLMo 2 7B preference dataset. Because DPO performance does not monotonically increase with more data (see Figure 23 ), we optimize the prompt distribution as a ratio within a set data budget and treat dataset size as a hyperparameter when training.

To determine our final preference-tuning prompt distribution, we begin with near-uniform random sampling 55 55 55 We decided early to truncate the number of Wildchat prompts to be at most 35% of the prompt mix. If you read Wildchat prompts for a month, you would too. of 100K examples as an empirically strong baseline prompt mix. We then perform ablations of prompt-domain subsets to determine the impact of preference pairs from each domain subset. Additionally, we perform experiments that pair 50K samples of our base mix with 50K samples from a given domain, allowing us to understand the effects of upsampling each prompt domain.

Notably, prompt-domain distributions do not consistently align with the contrast exhibited in the response pair and thus in improvements in the corresponding downstream evaluation domains. For example, upsampling code prompts led to the counter-intuitive effect of decreasing code benchmark performance (see Table 51 in the Appendix). For determining our final mix, we create nine candidate mixes based on expert intuition gained from our ablations, comparing these hand-crafted mixes against the uniform sampling baseline. Our final mix is determined empirically; we find that our hand-crafted mixes outperformed random sampling.

#### 5.3.3 Training

We follow the same training setup as Olmo 3 Think and sweep the same hyperparameters, namely learning rate and dataset size. We further sweep different length-control interventions by creating datasets with differing token cutoffs for length filtering. We select the best-performing checkpoint of each length budget and then select the final Olmo 3 Instruct -DPO checkpoint based on qualitative vibe tests and performance-vs-length analysis.

### 5.4 Reinforcement Learning with Dolci Instruct -RL

For our RL training stage, we modify the pool of prompts from Dolci Think RL (Section § 4.4.2 ) by 1) utilizing less challenging datasets in the math and code domains, and 2) skipping the offline difficulty filtering, as our instruct model focuses more on general instruction following rather than complex reasoning.

#### 5.4.1 Training