Olmo 3 — part 10/19

Cleaned-text code points [140386, 155608). Bibliography omitted here; source.md includes it.

• No standard deviation normalization : When calculating advantage, we do not normalize by the standard deviation of the group, following Liu et al. (2025b) . This removes a difficulty bias, where questions with low standard deviation in their rewards (e.g., too hard or too easy) have their advantages significantly increased by the normalization term.

##### OlmoRL formulation

Our final objective function includes a token-level loss , truncated importance sampling , clip-higher , and no standard deviation in the advantage calculation : 𝒥 ⁡ ( θ ) \displaystyle\mathcal{J}(\mathrm{\theta}) = 1 ∑ i = 1 G | y i | ​ ∑ i = 1 G ∑ t = 1 | y i | min ⁡ ( π ⁡ ( y i , t ∣ x , y i , < t ; θ old ) π vllm ​ ( y i , t ∣ x , y i , < t ; θ old ) , ρ ) ​ min ⁡ ( r i , t ​ A i , t , clip ⁡ ( r i , t , 1 − ε low , 1 + ε high ) ​ A i , t ) , \displaystyle={\color[rgb]{0.0664,0.3203,0.3398}\frac{1}{\sum_{i=1}^{G}|y_{i}|}\sum_{i=1}^{G}\sum_{t=1}^{|y_{i}|}}{\color[rgb]{0.9414,0.3203,0.6133}\min\!\Big(\frac{\pi\!\left(y_{i,t}\mid x,y_{i,<t};\theta_{\mathrm{old}}\right)}{\pi_{\mathrm{vllm}}\!\left(y_{i,t}\mid x,y_{i,<t};\theta_{\mathrm{old}}\right)},\rho\Big)}\min\Bigl(r_{i,t}\,{\color[rgb]{0.6953,0.1055,0.9102}{A}_{i,t}},\;\operatorname{clip}\bigl(r_{i,t},\;1-\varepsilon_{\mathrm{low}},\;1+{\color[rgb]{0.0586,0.7969,0.5508}\varepsilon_{\mathrm{high}}}\bigr)\,{\color[rgb]{0.6953,0.1055,0.9102}{A}_{i,t}}\Bigr), (1) where r i , t = π ⁡ ( y i , t ∣ x , y i , < t ; θ ) π ⁡ ( y i , t ∣ x , y i , < t ; θ old ) r_{i,t}=\frac{\pi\left(y_{i,t}\mid x,y_{i,<t};\theta\right)}{\pi\left(y_{i,t}\mid x,y_{i,<t};\theta_{\mathrm{old}}\right)} , ε low \varepsilon_{\mathrm{low}} and ε high {\color[rgb]{0.0586,0.7969,0.5508}\varepsilon_{\mathrm{high}}} are the clipping hyperparameters. Here, y i ∼ π vllm ( ⋅ ∣ x ; θ old ) y_{i}\sim\pi_{\text{vllm}}(\cdot\mid x;\theta_{\mathrm{old}}) and π vllm ( ⋅ ∣ x ; θ old ) \pi_{\text{vllm }}(\cdot\mid x;\theta_{\mathrm{old}}) are the token probabilities returned from vLLM, ρ \rho is the truncated importance sampling cap value ( Yao et al., 2025 ) , and the advantage A i , t A_{i,t} for the t t -th token t t in the response y i y_{i} is calculated within the group G G based on the relative reward of the outputs inside each group: A i , t = ( r ⁡ ( x , y i ) − mean ⁡ ( { r ⁡ ( x , y i ) } i = 1 G ) ) . \displaystyle{\color[rgb]{0.6953,0.1055,0.9102}{A}_{i,t}=\left(r\left(x,y_{i}\right)-\operatorname{mean}\left(\left\{r\left(x,y_{i}\right)\right\}_{i=1}^{G}\right)\right)}. (2)

r ⁡ ( x , y i ) r\left(x,y_{i}\right) is the reward score returned by the corresponding verifier. Our hyperparameters for various runs are in Appendix Table 49 .

##### Verifiers

We extend verifiable rewards beyond math domains from OLMo 2 to include general domains. For each domain we use a different custom verifier (see Figure 16 ): • Math We use a rule-based verifier that performs basic normalization and compares with a reference answer with SymPy to determine answer correctness. The verifier returns 1 if the answer is determined the same as the reference answer and 0 otherwise.

• Code We use a test-case based verifier that runs a set of test cases over the response. We experiment with (a) using the percentage of passed test cases as the reward and (b) returning 1 when the response passes all test cases and 0 otherwise. 35 35 35 Code execution When performing RL on code environments, we need to actually execute the generated code against test cases to calculate our rewards. We use AWS Lambda to do so. Using a distributed cloud function approach ensures that verification does not block the trainer process, and allows us to scale seamlessly. Many test case suites, such as those present in SYNTHETIC-2 ( PrimeIntellect, 2025 ) , contain test cases designed to penalize programs with poor time complexity, and running these tests can exceed hundreds of MBs for a single program, exceeding the resources of our local machines.

• Instruction-following We pass the response through a set of functions that check adherence to a series of constraints from the prompt. A reward of 1 is assigned if all constraints are satisfied, and 0 otherwise.

• Chat—reference For tasks with a ground-truth response, we pass the response to an LM judge to compare the model’s response against a provided reference answer, and ask the judge to give a score in [0, 1] based on the quality of the response.

• Chat—open-ended We pass the response to an LM judge and ask the judge to give a score in [0, 1] based on the quality of the response without any reference answer. 36 36 36 Unless otherwise stated, for an LM judge we host Qwen3 32B ( Yang et al., 2025a ) with thinking mode turned off using vLLM ( Kwon et al., 2023 ) , and allow a max input prompt of 32768 tokens while only allowing a response length of 2048 tokens. We provide the judge prompts in Figure 40 in the appendix. We additionally experimented with puzzle problem (checking if a puzzle solution is correct relative to a reference answer) and length-control ( Aggarwal and Welleck, 2025 ) verifiers, but did not find it useful for downstream performance.

#### 4.4.2 Dolci-Think-RL : Curating a State-of-the-art RLVR Dataset

We curate a large-scale and diverse dataset of roughly 100K samples across four domains: mathematics, coding, instruction following, and general chat to support robust RL on varied reasoning tasks while maintaining general utility. Each domain is associated with either a verifiable or non-verifiable reward signal (continuous or binary), ensuring that every instance can be automatically checked for correctness or general quality (see Figure 16 ). For all domains we take particular care with the provenance and licensing of sources. We provide the size of each dataset subsection after sourcing, filtering, and mixing in Table 20 .

##### Step 1: sourcing prompts

In what follows, we will describe our data construction process.

• Math : We combine community-curated math problems, including Open-Reasoner-Zero ( Hu et al., 2025 ) , DAPO-Math ( Yu et al., 2025 ) , AceReason-Math ( Chen et al., 2025 ) , DeepScaler ( Luo et al., 2025b ) , KlearReasoner-MathSub ( Su et al., 2025c ) , and OMEGA ( Sun et al., 2025 ) covering a wide range of mathematical domains including algebra, combinatorics, number theory, and geometry.

• Coding To construct reinforcement learning (RL) data for code, we required pairs of (problem, test cases) . We curate a diverse set of prompts for coding problems, including AceCoder ( Zeng et al., 2025a ) , Klear-Reasoner Code ( Su et al., 2025c ) , Nemotron Post-training Code ( NVIDIA AI, 2025 ) , SYNTHETIC-2 code ( PrimeIntellect, 2025 ) , and Open-Code Reasoner ( Ahmad et al., 2025 ) . We use the Klear-Reasoner and SYNTHETIC-2 test cases directly. For the other datasets, we run prompts through the following synthetic data pipeline: (1) problem rewriting , (2) solution generation , and (3) test case generation . After generating these triplets (problem, solution, test cases), we executed all model-generated or rewritten test cases against the corresponding solutions and kept examples with solutions that passed more than 80% of test cases while removing failed test cases. The resulting filtered dataset provided high-quality (problem, test cases) pairs suitable for training and experimentation with RL methods for code. We use the AceCoder prompts in function completion format, while all other datasets are in stdio format. Details of each step in code data synthesis pipeline can be found in Appendix A.7.3 .

• Instruction-following We use the prompts from IF-RLVR ( Pyatkin et al., 2025 ) with up to 5 constraints, which are sampled from IFEval ( Zhou et al., 2023 ) and IFBench-Train ( Pyatkin et al., 2025 ) .

• General chat We sample our general chat instances from three sources: (a) Tülu 3 SFT ( Lambert et al., 2024 ) ; (b) the new WildChat-4.8M data 37 37 37 huggingface.co/datasets/allenai/WildChat-4.8M containing a broad spectrum of user-chatbot interactions on ambiguous requests, code-switching, topic shifts, political debates, and more; and (c) the Multi-subject-RLVR dataset ( Su et al., 2025b ) , consisting of college-level English questions and objective answers written by domain experts for examination purposes. For WildChat, we only sample from instances that are in English and do not require reasoning (such as math and code). For Tülu 3, we first rewrote samples using GPT-4.1 for better clarity and to extract reference answers from the SFT set. We then generated eight samples per prompt with a Qwen 2.5 7B model finetuned on OpenThoughts 2 and computed the F1 score between the reference answer and each response. We then removed all samples with average F1 score < 0.1 <0.1 and > 0.8 >0.8 . This removes both noisy and overly difficult samples. WildChat in particular has a high prevalence of role-playing and other character-based data. In order to balance the data, we filter any mention of a single character down to a maximum of 10 instances. 38 38 38 In our intermediate general dataset of 57,819 samples, we found the top characters were 1. Natsuki : 1284 appearances, 2. Monika : 1243, 3. Sayori : 1076, 4. Yuri : 957, 5. Sakura: 453, and 6. MC: 424. All others were at 60 or lower before filtering. We then finally performed some post-hoc manual filtering to remove code- and math-centric prompts.

##### Step 2: offline difficulty filtering

As stated previously, to improve the sample efficiency of RL for our reasoner model, we generate eight rollouts for each prompt from the initial checkpoint of the model we train (e.g., if starting from the DPO-trained model, we generate from the DPO checkpoint). We then remove all samples that the model easily solves (that is, those with a pass rate greater than 62.5%). We sample with a temperature of 1.0 and top-p of 1.0, matching how we sample during RL training. We used offline filtering for the 7B Olmo 3 Think to filter out RL problems that are too easy for our models’ training. For the 32B, we rely on active sampling, which fills RL batches only on samples with a non-zero GRPO group gradient, and re-using the 7B DPO-filtered data as the starting point for the model due to compute and time constraints.

##### Step 3: data mixing

When developing our data mixture and overall recipe, we found RL experiments were both long and compute-expensive, preventing us from ablating the full space of datasets and algorithmic choices. Instead, we established a pipeline in which: (a) we performed dataset-specific runs on an intermediate SFT checkpoint and observed downstream evaluation trends over the first 500-1000 RL steps; (b) focused on math domain training when testing new algorithmic changes; (c) periodically ran overall mixture experiments to ensure mixing was stable. When setting up our final run, we then took the most promising datasets, performed offline filtering, and carefully mixed them to ensure higher-quality datasets were upweighted, and roughly equal amounts of data were used for each domain (with slightly more focus on math and instruction following, as training on these domains seemed the most effective in per-dataset runs). Additionally, we downsample certain subtasks from OMEGA that the model especially struggled with based on offline filtering results. 39 39 39 In particular, we downsample the following tasks by 50% after filtering: trans_integrations , logic_gridworld_rookmove , logic_puzzles_grid_chip , comp_grid_chips , comp_n_gon , arithmetic_matrix_svd , comp_parametric_intersection , comp_vertex_color . We used this pipeline to develop an RL mixture for the 7B model, and then used the same data mixture for the 32B model due to compute and time constraints.

For our Olmo 3 Think 7B training run, we used an initial version of our infrastructure without pipelineRL or truncation importance sampling, which took approximately 15 days. We later replicated the same run with our newer infrastructure, achieving similar performance in just 6 days of training.

#### 4.4.3 OlmoRL Infrastructure in Open Instruct

We made substantial improvements to our reinforcement learning infrastructure to handle longer sequences and faster overall throughput. In RL with LLMs, the key technical challenge for finetuning models that generate long sequences is managing inference—also called the rollouts. For our final models, we generated rollouts with a maximum size of 32K tokens in length, averaging more than 10K tokens (for the reasoner models). Inference dominated our computational costs, using 8 H100 nodes for training and 20 nodes for inference for the 32B OlmoRL reasoner model. Given the cost of autoregressive inference, our learner spends 75% of the time waiting for data, so in terms of GPU utilization, we use approximately 5x as much for inference as for training. In fact, we use the minimal possible sharding configuration to fit the learner in memory and do not prioritize speed at all, unlike in the supervised learning setting. For the 7B reasoner model, where we have less memory pressure on the learner, the situation was more dramatic, as we used 7 nodes for inference and only 2 for the learner. Given the similarly low utilization of the learner, we used approximately 14x as much compute for inference as for training. We suspect that we have a suboptimal sharding configuration for the 32B learner and expect that we could do better in future work.

##### Fully asynchronous training

Shown in 17(a) , we employ an off-policy asynchronous RL setup ( Noukhovitch et al., 2024 ) featuring a centralized learner distributed across multiple nodes via DeepSpeed ( Rasley et al., 2020 ) and a large pool of actors, each running an independent vLLM ( Kwon et al., 2023 ) instance. The learner produces prompts that are queued and dispatched to the actors, which execute the prompts, interact with the environment, and return results through a results queue that the learner uses to update the model parameters. 40 40 40 For the 7B training runs, we use a single GPU for each actor and scale generation via data parallelism. The RL setup would be familiar to readers of Horgan et al. (2018) or Silver et al. (2017) . For 32B, we use one node per actor and then similarly further scale via data parallelism. Due to the variance in completion length, a long time delta can emerge between completions in an individual batch of RLVR. The guiding principles to mitigate this issue are to make efficient use of resources (avoiding idling) and to make processes asynchronous. 41 41 41 For one of our main RL runs, which was broadly representative of what we experienced across all of our runs, each training step averaged 1000 seconds, of which 125 seconds was spent running training. Each batched completion generation took 1000 seconds. As we overlap generation and training ( Noukhovitch et al., 2024 ) , the bottleneck is entirely generation. Consequently, significant engineering resources were spent improving the way generation is handled, where we could continue to use the training code used in OLMo 2 , as we would need to speed up generation by > 8 × >8\times for that to be a bottleneck.

##### Continuous batching