Olmo 3 — part 18/19
Cleaned-text code points [265504, 279647). Bibliography omitted here; source.md includes it.
Given the varied quality of real-world queries, GPT-5 is employed to rate each query drawn from existing open-access benchmarks on a five-point scale assessing (i) whether it calls for comprehensive long-form responses, (ii) factual verifiability, and (iii) the degree of search required. Only queries scoring 4 or 5 on these criteria are retained. We then use an agent equipped with web search and browsing via the Serper API, and scientific snippet retrieval via ASC to generate tool-use trajectories for these queries. This agent is instructed with tool specifications and step-by-step search instructions, resulting in detailed trajectories containing both tool calls and environment outputs. We then filter out trajectories that yield incorrect answers (where ground truth is available), and only keep trajectories that adhere to the expected output format. Additionally, since the environment outputs for the webpage-fetching tool of the Serper API are quite long (typically entire webpages), we used GPT-5 to summarize the content of the web pages and only retained the summaries in the training data. ##### Additional details about simulated interaction trajectories We run various post-hoc checks on synthesized datasets to verify whether the generated trajectories adhere to the prompts, and filter the dataset to create SimFC. We filter out trajectories where the function calls include functions not part of the presented APIs. Our data-synthesis prompts explicitly target multi-turn, multi-step, parallel function calls (i.e., multiple calls per assistant turn) and refusals, and we filter out the trajectories that do not conform to such requirements specified in the prompts. #### A.7.3 Coding Data Synthesis Pipeline To construct reinforcement learning (RL) data for code, we required pairs of (problem, test cases) . We curate a diverse set of prompts for coding problems, including AceCoder ( Zeng et al., 2025a ) , Klear-Reasoner Code ( Su et al., 2025c ) , Nemotron Post-training Code ( NVIDIA AI, 2025 ) , SYNTHETIC-2 code ( PrimeIntellect, 2025 ) , Open-Code Reasoner ( Ahmad et al., 2025 ) . We use the klear-reasoner and SYNTHETIC-2 test cases directly. For the other datasets, we run prompts through the following synthetic data pipeline: • Problem rewriting. Given a coding problem, we first prompted GPT-4.1 to rewrite the description so that it either (a) included a function signature, or (b) explicitly specified that the solution should read from and write to standard input/output (stdio) • Solution generation. GPT-4.1 was then prompted to provide a corresponding solution. Depending on the problem type, this was either a Python function matching the given signature, or a program reading from and writing to stdio. When the original problem source included a reference solution, we included it in the prompt • Test case generation. GPT-4.1 was further prompted to generate test cases in the appropriate format (function-based or stdio-based) #### A.7.4 Dolci Instruct DPO Details ##### DPO prompt mixing See Table 51 for prompt mixing experiment results. ##### Model pool for LLM-judged pairs To create the GPT-judged subset of Dolci Instruct DPO , we generate completions on our prompt pool with the following models: gpt-oss-20B, gpt-oss-120B ( Agarwal et al., 2025 ) , GPT-4.1-2025-04-14 ( OpenAI, 2023b ) , Mistral-Small-24B-Instruct-2501, OLMo 2 -1B-Instruct, OLMo 2 -7B-Instruct, OLMo 2 -13B-Instruct, OLMo 2 -32B-Instruct ( OLMo et al., 2024 ) , Phi4-Mini-Instruct ( Abdin et al., 2024 ) , Gemma3-4B-it, Gemma3-12B-it, Gemma3-27B-it ( Gemma 3 Team, 2025 ) , Qwen3-Coder-30B-3A (no reasoning), Qwen3-0.6B (no reasoning), Qwen3-1.7B (no reasoning), Qwen3-4B (no reasoning), Qwen3-8B (no reasoning), Qwen3-14B (no reasoning), Qwen3-32B (no reasoning), Qwen3-30B-3A (no reasoning) ( Yang et al., 2025a ) , QwQ-32b ( Qwen Team, 2025 ) , Yi-9B, and Yi-34B ( Young et al., 2024 ) . For each prompt, we sample four model completions and judge them via a GPT-4.1 judge with the UltraFeedback judge prompts 70 70 70 We ran initial experiments employing a GPT-5 judge, but results indicatedthat the GPT-4.1 judge is better. ( Lambert et al., 2024 ; Cui et al., 2023 ) . To enforce a meaningful delta between chosen and rejected responses, we enforce our judge pipeline to sample responses from exactly two of the following smaller and/or previous generation models which show lower overall performance: OLMo 2 -1B-Instruct, OLMo 2 -7B-Instruct, Yi-9B, Yi-34B, Phi4-Mini-Instruct, Qwen3-0.6B (no reasoning), Qwen3-1.7B (no reasoning). Without this intervention (i.e. sample four models from the pool to judge at random), we would have an approximately 33% chance of sampling at least 2 weak models out of our 4 samples from our model pool for judgment, providing limited contrast in preference pairs. We binarize into preference pairs by selecting the worst response out of the four to be rejected, and the best as chosen. ### A.8 Post-Training Additional Evaluation Details #### A.8.1 General Evaluation Settings For post-training, we focus exclusively on generative evaluations, in which we generate completions until a max length is reached or eos token is generated (as opposed to multiple-choice-based evaluations used in pretraining), better matching real-world downstream usage. Following DeepSeek R1 report ( Guo et al., 2025 ) and Nvidia Nemotron ( Adler et al., 2024 ) we use a sampling temperature of 0.6 and top-p of 0.95. 71 71 71 We find that both thinking models degenerate quickly when evaluated with low temperatures (as used in OLMo 2 ), while instruction models can be evaluated at this higher temperature. We strip thinking traces from the answer text when generated. We account for the variance this induces in smaller benchmarks (e.g. AIME, which is made up of 30 questions) by taking multiple samples and reporting the overall average performance. For QA tasks (e.g. BBH, MMLU), we create a unified set of ‘Olmo 3’ regexes for answer extraction, covering a wide variety of potential answer templates. We additionally update AlpacaEval 2 Length Controlled (LC) ( Dubois et al., 2024 ) to use GPT-4.1 as a judge instead of the original GPT-4-Turbo ( OpenAI, 2023b ) both to increase the reliability of the evaluation and to save ∼ \sim 90% of inference costs. Importantly, our evaluation settings are unified across thinker and instruct models , simplifying our evaluation development process. ##### Is AlpacaEval useful? Certainly! AlpacaEval , and similar evaluations, such as ChatBotArena ( Zheng et al., 2023 ) , MT-Bench ( Zheng et al., 2023 ) , Arena-Hard ( Li et al., 2024c ) , etc. are established as crucial benchmarks for the AI industry . Let’s delve into the pros and cons of AlpacaEval: It’s not a broken evaluation, it’s a trade-off . It’s well established that most people enjoy reading language model completions that have a bit of flair to them. In fact, the style of bold, lists, etc. can be very helpful when skimming information . It just can often go over the top—such as when too many emoji’s are included! Pro: Ease-of-reading and flair Con: Over-optimized style We’re incentivized to maximize the benchmark—even if we don’t like it . As a smaller lab , we need to work hard to put our models on the map! We don’t love the style of completions from models scoring high on these benchmarks, but we derive so much benefit from the attention it attracts . Pro: Simple comparison to known standards Con: Imperfect performance signal There aren’t many better options! There are just so few evaluations that test a model’s ability to chat with the users reliably —and we need to serve the most common use case if we want adoption . More diversity of benchmarks , such as alternatives like multi-turn and instruction following, are slowly helping out understanding. Pro: Common adoption Con: Low diversity in chat evaluations Bonus: There’s something poetic about having LLMs judge LLMs. In summary, we need evaluations like this to make sure the model is behaving as expected . When it comes to balancing style and benchmarks , at the end of the day, no-one’s perfect—not even us . OLMo 3 7B Think Baselines Benchmark SFT DPO Final Think OpenThinker3 7B Nemotron Nano 9B v2 DS-R1 Qwen 7B Qwen 3 8B Qwen 3 VL 8B Think OR Nemotron 7B DoAnythingNow 19.3 19.6 23.4 1.8 56.7 34.3 53.1 83.0 2.3 HarmBench 67.8 72.7 75.4 26.7 69.4 50.7 74.0 81.9 20.0 TrustLLM-JailbreakTrigger 64.8 65.2 72.0 2.9 62.6 50.1 56.7 77.0 6.9 WildJailbreak-Test Harmful 23.4 27.5 39.0 0.3 28.7 4.5 12.3 38.6 0.5 WildJailbreak-Test Benign 99.1 98.5 98.8 99.2 97.3 98.0 99.7 98.0 97.1 WildGuard-Test 90.2 93.9 93.8 48.8 88.4 69.2 82.9 93.0 42.6 XSTest 91.6 91.6 90.9 59.5 92.5 68.4 87.2 94.2 61.0 BBQ Accuracy 86.6 84.8 89.2 80.5 92.0 78.0 91.8 86.6 82.6 BBQ Bias - Ambig. 7.3 8.4 6.5 11.3 5.8 9.4 5.5 8.9 7.1 BBQ Bias - Disambig. 1.7 1.1 1.7 2.4 0.7 2.4 1.5 1.0 2.3 StrongReject 74.8 75.5 79.0 56.7 85.6 72.4 73.4 82.8 58.3 Toxigen 100 99.9 100 97.4 100 99.7 100 99.9 86.4 WMDP 46.4 43.4 42.7 45.5 38.3 55.9 34.9 38.7 51.8 OLMo 3 7B Instruct Baselines Benchmark SFT DPO Final Instruct Qwen 3 8B (No Thinking) Qwen 3 VL 8B Inst Qwen 2.5 7B OLMo 2 7B Inst Apertus 8B Inst Granite 3.3 8B Inst DoAnythingNow 90.0 82.9 75.2 81.2 53.8 59.0 92.0 43.1 36.8 HarmBench 87.7 94.3 94.9 74.2 84.6 80.1 88.8 79.3 86.3 TrustLLM-JailbreakTrigger 84.8 85.2 79.2 76.8 76.1 63.8 85.8 55.4 63.6 WildJailbreak-Test Harmful 80.9 72.5 69.1 21.2 37.4 13.4 76.8 43.0 66.4 WildJailbreak-Test Benign 88.1 96.4 98.0 99.3 97.3 99.3 96.8 94.4 84.7 WildGuard-Test 98.8 99.8 99.6 86.8 91.0 87.5 99.2 89.9 93.8 XSTest 91.3 93.1 93.2 91.3 93.2 93.8 93.9 90.1 89.9 BBQ Accuracy 74.3 75.5 79.0 87.6 87.9 88.5 74.6 73.4 68.8 BBQ Bias - Ambig. 9.1 9.3 8.6 8.5 7.8 6.8 9.4 7.0 4.5 BBQ Bias - Disambig 4.4 3.4 2.7 1.8 -0.1 3.5 2.7 2.5 2.7 StrongReject 93.5 89.2 88.1 83.5 85.3 78.2 89.4 76.9 82.0 Toxigen 100.0 100.0 100.0 100.0 100.0 100.0 100.0 100.0 100.0 WMDP 47.2 45.3 45.5 35.8 35.6 41.3 51.6 48.5 46.9 #### A.8.2 Safety Evaluations Overview The safety evaluations that were tested upon during training runs and whose average was reported earlier were the same set from OLMo 2 ( OLMo et al., 2024 ) and Tülu 3 ( Lambert et al., 2024 ) . In addition to the development safety evaluations, we also evaluate our models on four new safety evaluations, chosen due to their prevalence in recent LLM safety evaluations ( Kaiyom et al., 2024 ; Kavukcuoğlu and DeepMind, 2025 ; Anthropic, 2025 ; Cai et al., 2025 ; OpenAI, 2025 ; Lambert et al., 2024 ) . ##### Development safety evaluations We include HarmBench ( Mazeika et al., 2024 ) , DoAnythingNow ( Shen et al., 2024 , DAN;) , XSTest ( Röttger et al., 2023 ) , WildGuard-Test ( Han et al., 2024 ) , WildJailbreak-Test ( Jiang et al., 2024 ) , and TrustLLM-JailbreakTrigger ( Huang et al., 2024a ) . ##### Unseen safety evaluations We further evaluated on four held-out safety benchmarks: Toxigen ( Hartvigsen et al., 2022 ) , StrongReject ( Souly et al., 2024 ) , Weapons of Mass Destruction Proxy ( Li et al., 2024b , WMDP;) , and Bias Benchmark for QA ( Parrish et al., 2022 , BBQ;) . ##### Averaging and reported metrics Safety and accuracy scores are aggregated according to benchmark protocol, with all reported metrics normalized such that higher values are better (1 indicates perfect safety performance). Specifically, we report the average of: refusal accuracy , i.e., inverted ASR (Attack Success Rate), for DoAnythingNow, Harmbench, Wildguard, TrustLLM-JailbreakTrigger, Toxigen, and StrongReject; accuracy for XSTest and BBQ; the average of inverted ASR for Wildjailbreak (harmful) and ASR for Wildjailbreak (benign); and inverted accuracy (i.e., error rate) for WMDP. For the safety benchmarks, models were evaluated with a top-p of 0.95 and sampling temperature of 0.7. We explain all of the evaluations in more detail below: • HarmBench ( Mazeika et al., 2024 ) evaluates models’ refusal to comply with a diverse suite of harmful prompts, distributed across both functional and semantic categories. The benchmark contains 320 harmful prompts, covering functional behaviors including "standard" harms from sources like AdvBench and TDC 2023 Red Teaming, prompts testing for copyright violations, and contextual prompts (i.e., prompts combining an instruction with a context such as a document). 72 72 72 In our implementation, the context string is merged into the input prompt for contextual examples. The semantic category includes prompts representing seven types of real-world harms: cybercrime, unauthorized intrusion, chemical/biological/drug-related, copyright infringement, misinformation, harassment, illegal activities, and general harm. Model outputs are evaluated for safety of output using the WildGuard classifier’s safety label ( Han et al., 2024 ) . • DoAnythingNow ( Shen et al., 2024 , DAN;) tests models’ robustness to the well-known DAN jailbreak framework by pairing DAN-style jailbreak templates with harmful behaviors adapted from HarmBench. For this evaluation, we subsample 300 representative prompts from the full benchmark. Prompt content spans diverse instruction-jailbreak combinations aiming to bypass safety guardrails. We compute refusal accuracy 73 73 73 Refusal accuracy corresponds to 1-ASR (attack success rate). using the WildGuard classifier’s refusal label ( Han et al., 2024 ) to assess whether model outputs refuse or comply with the harmful instructions. • XSTest ( Röttger et al., 2023 ) measures models’ over-refusal tendencies, i.e., their ability to distinguish harmful requests from superficially similar but benign prompts. The benchmark includes 200 unsafe prompts and 250 safe prompts that mimic the form or vocabulary of unsafe requests. Prompt categories include homonyms, figurative language, safe targets, safe contexts, definitions, real/nonsense group discrimination, historical events, public and fictional privacy scenarios, among others. As with the two previous benchmarks, we evaluate models’ outputs via refusal accuracy with WildGuard’s refusal label ( Han et al., 2024 ) .