Olmo 3 — part 9/19

Cleaned-text code points [124702, 140386). Bibliography omitted here; source.md includes it.

To curate Dolci Think SFT , we compile a large collection of prompts across a diverse set of skills from other open efforts (e.g., Guha et al., 2025a ; PrimeIntellect, 2025 ), substantially filter them, and synthetically generate reasoning traces for their completions. An overview of the Dolci Think SFT data mix is shown in Table 17 and is described below:

##### Step 1: sourcing prompts and generating reasoning traces

• Math We source prompts from the math subsets of OpenThoughts3 ( Guha et al., 2025a ) and SYNTHETIC-2 ( PrimeIntellect, 2025 ) . For OpenThoughts3 prompts, we use all the available math prompts (maintaining the 16X repetition from the original) and the available reasoning traces with complete solutions. For incomplete traces, we generate full reasoning chains and solutions using QwQ-32B, the original model used for the completions, and the same generation settings as OpenThoughts3, except up to 32K tokens instead of the original 16K. We discard any examples that are still incomplete after regenerating. For SYNTHETIC-2, we take completions directly from the verified subsection.

• Code We collect code prompts from different sources and generate completions for them. To create Dolci Think Python Algorithms, we source prompts from AceCoder ( Zeng et al., 2025a ) , the Python subset of The Algorithms ( The Algorithms, 2025 ) , Llama Nemotron Post-training ( Bercovich et al., 2025 ) , and OpenCodeReasoning ( Ahmad et al., 2025 ) , and then we generate up to 16 responses per prompt from QwQ-32B, which we filter for correctness using synthetically generated test cases from GPT-4.1. For OpenThoughts 3 code prompts, we downsample each prompt to at most 16 times and regenerate complete responses for all incomplete examples. We combine Dolci Think Python Algorithms with the code prompts from OpenThoughts3, downsample them to 16 repetitions, and regenerate completions for incomplete ones.

• Chat & safety We source chat prompts from both the Tülu 3 ( Lambert et al., 2024 ) subset of WildChat ( Zhao et al., 2024a ) , as well as WildChat prompts not used during Tülu 3, and the Tülu 3 subset of OpenAssistant ( Köpf et al., 2024 ) . For safety, we reuse safety prompts used during Tülu 3. We then generate reasoning traces and completions from DeepSeek R1 ( Guo et al., 2025 ) .

• Precise instruction following We source precise IF prompts from the overall Tülu 3 mix with additional verifiable constraints added from Pyatkin et al. (2025) . We also regenerate Persona IF prompts as in Tülu 3, but with personas sourced from Meyer and Corneil (2025) . We then generate responses for each prompt using QwQ-32B, and we verify responses using verifiers associated with each constraint, keeping only the correct responses.

• Science & other We source science prompts from the OpenThoughts3 science subset. For other data sources, we include the TableGPT ( Zha et al., 2023 ) subset in Tülu 3 for data transformation and Aya ( Singh et al., 2024 ) for chat and basic multilinguality. We regenerate incomplete responses in OpenThoughts3 as we did for the math and code subsets, and we generate responses with reasoning chains for the other datasets using DeepSeek R1.

##### Step 2: filtering

We perform extensive filtering on the data we have collected and generated. • Heuristic filtering We filter out examples with (1) non-commercial or unclear licenses, (2) incomplete reasoning chains, (3) domain-specific inaccuracies (i.e., verifying the constraint-adherence of instruction-following data or executing test cases against model completions for code), (4) mentions of other model developers and date cutoffs, (5) excessive repetition, and (6) an excessive number of Chinese characters or Chinese political values reflected in reasoning chains.

• Topic filtering We classify our dataset by topic using the OpenAI query taxonomy ( Chatterji et al., 2025 ) , and find that filtering out and downsampling topics irrelevant to our model (e.g., requests to generate images or excessive basic greetings) from WildChat qualitatively improves model behavior. 31 31 31 To evaluate the impact of our filtering process, we manually created an internal benchmark to vibe test the model. See Appendix A.7.1 for detailed descriptions and links to filter scripts.

##### Step 3: data mixing

For data mixing, we follow a methodology similar to that described in the midtraining section (Section § 3.5 ) for parallel data collection, adhering to shared standards for data mixing and conducting multiple rounds of integration testing. More specifically, we conduct careful experiments using a small “base” mix, consisting of 100K examples taken from our extended OpenThought 3 dataset. We found that this base mix was performant enough on key reasoning benchmarks to serve as a strong baseline, while saving substantial amounts of compute versus training on the full mix. We then train individual models on the base mix combined with up to 100K training examples (without upsampling) from each category to observe the impact on our evaluation suite. As shown in Table 18 , we generally find that each dataset is helpful on at least one evaluation, and so our final mix includes at least a portion of each dataset we tested.

##### Step 4: decontamination

We followed the recommended settings from the Tülu 3 Decontamination Procedure and toolkit ( Lambert et al., 2024 ) to filter out the portions of all post-training data (all three stages) that matched the evaluation sets. We used n-gram matching with 8-grams and an overlap threshold of 0.5 (i.e., at least 50% of the n-grams in the test instance match a training instance) for filtering. We developed additional heuristics to mitigate false positives: (1) we ignored matches of task-irrelevant chunks of text, e.g., common generic phrases, with the irrelevance determined per task based on manual inspection; (2) particularly in math datasets, we ignored matches of n-grams where most of the tokens are of length 1 (typically math symbols).

#### 4.2.2 Training

For supervised finetuning, we switch from Open Instruct to OLMo-core, resulting in an 8 × 8\times increase in training throughput. See Appendix A.6.1 for more information about our training settings and hyperparameters. We train all models for two epochs to avoid overfitting, and perform a learning-rate sweep to select the best candidate checkpoints based on our evaluation suite. We then test each candidate checkpoint with a series of qualitative “vibe-test” questions to inform our final checkpoint selection. Finally, we explore model souping ( Wortsman et al., 2022 ; Morrison et al., 2024 ) , and our final thinking SFT checkpoint is a linearly weighted merge of two checkpoints trained with different learning rates, merged with mergekit ( Goddard et al., 2024 ) .

### 4.3 Preference Tuning with Delta Learning

Prior work in general post-training has positioned preference tuning primarily as a means to improve alignment with human values and preferences ( Lambert et al., 2024 ; Lambert, 2025 ) . Hence, most recent efforts in building capability-oriented thinking models ( Guha et al., 2025a ; Ahmad et al., 2025 ) have not incorporated preference tuning (one exception is SmolLM3; Bakouch et al. (2025) ). We rethink preference tuning as a stage of contrastive learning that drives capability gains beyond what SFT alone can provide. We introduce Dolci Think DPO , a preference dataset containing completion pairs with clear capability deltas. We leverage these relative contrasts to enhance the model’s reasoning capabilities via preference optimization, extending the ideas from Delta Learning ( Geng et al., 2025 ) .

In particular, we find that further supervised finetuning on thinking traces generated by Qwen3 32B (one of the few open-thought models) outright hurts the performance of Olmo 3 Think SFT , indicating that we are approaching saturation on learning from imitation. To extract a useful training signal out of these now-ineffective completions, we apply Delta Learning’s principle by pairing these completions with even worse responses ( Geng et al., 2025 ) ; minimizing the quality of the rejected completions (thus increasing the quality delta) yields a useful contrastive signal for preference tuning.

With these insights in mind, we construct Dolci Think DPO , which we use to improve the model’s performance across a wide range of benchmarks. We use Direct Preference Optimization (DPO) ( Rafailov et al., 2024 ) for training with pairwise data. Details of DPO training are provided in Appendix A.6.2 .

##### Delta Learning

The intuition behind delta-learning is that the quality of preference data depends primarily on the quality of the delta between chosen and rejected responses rather than the quality of either response individually. By constructing preference pairs ( x , y c , y r ) (x,y_{c},y_{r}) that exhibit capability-relevant contrasts with y c ≻ y r y_{c}\succ y_{r} , tuning to prefer y c y_{c} over y r y_{r} can improve the model even when supervised finetuning on y c y_{c} would not help or even actively hurt ( Geng et al., 2025 ; D’Oosterlinck et al., 2025 ; Kim et al., 2025 ) .

#### 4.3.1 Dolci Think -DPO: Preference Data Creation

To construct Dolci Think DPO , we compile a large pool of prompts covering a wide range of datasets and skills (see Table 19 ) and synthesize chosen and rejected responses to exhibit capability deltas. Following the delta-learning heuristic ( Geng et al., 2025 ) , for each prompt x x , we simply decode a chosen completion y c y_{c} from one model (Qwen 3 32B, thinking) and a rejected completion y r y_{r} from an overall weaker model (Qwen 3 0.6B, thinking) to construct a consistent contrast. 32 32 32 The UltraFeedback-style LLM-judge preference pipeline employed in OLMo 2 and Tülu 3 assumes access to a diverse pool of models to construct preference pairs with useful contrasts; however, there are few open-thought thinking models available to construct such pairs, rendering the OLMo 2 pipeline less ideal for this setting. Our Dolci Instruct DPO dataset does benefit from model-pool diversity; we are able to further supplement our delta-learning heuristic data with LLM-judged data in Dolci Instruct DPO to yield mutually complementary gains (Section § 5.3.1 ).

##### Step 1: sourcing prompts and contrastive completions

Olmo 3 Think focuses on reasoning capabilities; we thus construct pairs that exhibit a delta in reasoning quality by pairing model completions from models of differing reasoning capability ( Geng et al., 2025 ; Bakouch et al., 2025 ; Kim et al., 2023 ) . Our prompt pool is derived from the Dolci Instruct SFT dataset supplemented with the DaringAnteater ( Wang et al., 2024b ) and UltraFeedback ( Cui et al., 2023 ) subsets from the OLMo 2 7B preference dataset.

##### Step 2: filtering

We apply topic filtering and heuristic model-identity filtering as described from the SFT stage (Section § 4.2.1 ) to all chosen responses. We leave rejected responses unfiltered with the intuition that an incorrect rejected response may elicit a useful contrast. We further decontaminate all prompts against our evaluation suites.

##### Step 3: mixing

Experimentation with long reasoning traces is significantly more expensive than with non-thinking completions. To obtain the final mix of prompts for Dolci Think DPO , we leverage mixing experiments conducted on prompts with non-thinking completions (see Section § 5 for details). Specifically, we select the three best-performing prompt distributions from our Olmo 3 Instruct experiments and generate chosen and rejected responses for these prompts using the thinking versions of the Qwen models to elicit a delta in reasoning quality. We choose the empirically best-performing mix during our experiments as our final DPO data pool. 33 33 33 Our Dolci Instruct DPO dataset includes additional contrastive pairs, which we obtain through careful experimental analysis. Refer to Section § 5.3.1 for more details.

#### 4.3.2 Training

We train all models for one epoch following previous work ( Lambert et al., 2024 ) , sweeping learning rate and dataset size to identify the best candidate checkpoints based on our evaluation suite. Dataset size is an important hyperparameter, as we observe that early stopping is important for performant preference tuning; please see our data mixing experiments on our Instruct model (Section § 5.3.2 ) for our motivating results. Beyond our evaluation suite, we further inspect each checkpoint via the same “vibe-tests” as in SFT training to qualitatively assess model behavior. See Appendix A.6.2 for full training settings.

### 4.4 Reinforcement Learning with OlmoRL : The Cherry on Top

The third stage of post-training is reinforcement learning with a mixture of verifiable and LM-judge rewards across a variety of domains. We introduce OlmoRL , which includes our algorithm and closely intertwined engineering infrastructure to address challenges for reinforcement learning with long reasoning traces, extending RLVR to include a wider variety of verifiable tasks. We also release Dolci-Think-RL —a large-scale and diverse dataset of roughly 100K prompts across four domains: mathematics, coding, instruction following, and general chat—to support robust reinforcement learning on varied reasoning tasks while maintaining general utility. Next, we describe the RL algorithmic details (§ 4.4.1 ), the Dolci Think RL dataset (§ 4.4.2 ), and finally OlmoRL infrastructure in Open Instruct (§ 4.4.3 ).

#### 4.4.1 OlmoRL Algorithmic Details

Our reinforcement learning stage is powered by OlmoRL , an approach that builds on Group Relative Policy Optimization (GRPO) ( Shao et al., 2024 ) and integrates a number of recent algorithmic advances. In particular, we adopt improvements from DAPO ( Yu et al., 2025 ) and Dr GRPO ( Liu et al., 2025b ) , among others ( Yao et al., 2025 ; Piché et al., 2025 ) . At its core, the objective of RLVR is to maximize the expected reward of a model-generated response y y given the prompt x x , where a verifier checks whether the response y y matches the ground-truth answer associated with x x .

We make the following improvements 34 34 34 We experimented with additional changes (e.g., overlong filtering), but did not find these gave consistent improvements. over vanilla GRPO: • Zero gradient signal filtering : We remove groups of instances whose rewards are all identical (i.e., a batch with zero standard deviation in their advantage) to avoid training on samples that provide zero gradient, similar to DAPO ( Yu et al., 2025 ) .

• Active sampling : We maintain a consistent batch size in spite of zero gradient filtering with a novel, more efficient version of dynamic sampling ( Yu et al., 2025 ) , see Section § 4.4.3 for details.

• Token-level loss : We use a token-level loss to normalize the loss by the total number of tokens across the batch ( Yu et al., 2025 ) , rather than per-sample to avoid a length bias.

• No KL loss : We remove the KL loss as a common practice ( GLM-4.5 Team et al., 2025 ; Yu et al., 2025 ; Liu et al., 2025b ) as it allows less-restricted policy updates, and removing it does not lead to over-optimization or destabilized training.

• Clip higher : We set the upper-bound clipping term in the loss to a slightly higher value than the lower bound to enable larger updates on tokens, as proposed by Yu et al. (2025) .

• Truncated importance sampling : To adjust for differences between log probabilities from the inference and training engines, we multiply the loss by the truncated importance sampling ratio, following Yao et al. (2025) .