2 OLMo 2 Furious — part 5/7

Cleaned-text code points [62408, 78173). Bibliography omitted here; source.md includes it.

To adapt OLMo 2 to downstream generative tasks, we follow the Tülu 3 recipe ( Lambert et al., 2024 ) with an increased focus on permissive licenses and suitable adjustments to hyperparameters. The Tülu 3 approach involves three phases of training: supervised finetuning (SFT), preference tuning with Direct Preference Optimization (DPO; Rafailov et al., 2024 ) and on-policy preference data, and finally Reinforcement Learning with Verifiable Rewards (RLVR). We find that all of the stages in the Tülu 3 Recipe easily translate to the OLMo 2 models. This section focuses on the development of our 7B and 13B models, where the 1B and 32B models followed very similar recipes.

Category Benchmark CoT # Shots Chat Multiturn ICL Metric Knowledge Recall MMLU ✓ 0 ✓ ✗ EM PopQA ✗ 15 ✓ ✓ EM TruthfulQA ✗ 6 ✓ ✗ MC2 Reasoning BigBenchHard ✓ 3 ✓ ✓ EM DROP ✗ 3 ✗ N/A F1 Math GSM8K ✓ 8 ✓ ✓ EM MATH ✓ 4 ✓ ✓ Flex EM Instruction Following IFEval ✗ 0 ✓ N/A Pass@1 (prompt; loose) AlpacaEval 2 ✗ 0 ✓ N/A LC Winrate Safety Tülu 3 Safety ✗ 0 ✓ N/A Average ∗

##### Supervised Finetuning (SFT)

The SFT training of OLMo 2-Instruct from Tülu 3 relies on selecting the highest-quality, existing instruction datasets and complementing them with scaled synthetic data for Supervised Finetuning based on the PersonaHub method ( Chan et al., 2024 ) . We develop two SFT mixes— tulu-3-sft-olmo-2-mixture which we used for our 7B and 13B models and tulu-3-sft-olmo-2-mixture-0225 which includes minor modifications and applied to our 1B and 32B models.

For tulu-3-sft-olmo-2-mixture , given that OLMo 2 is not trained for multilingual tasks, we experimented with removing all multilingual data from the SFT stage. When removing the entire Aya split and the multilingual samples of Wildchat from Tülu 3, we saw a degradation of ∼ 0.5 \sim 0.5 points on average, indicating that the Tülu 3 dataset is balanced and cannot be easily improved by removing irrelevant subsets. In total, this SFT mix contains 939,104 prompts.

For the 1B and 32B mix, tulu-3-sft-olmo-2-mixture-0225 , we further filtered out instructions that included mentions of a date cutoff from the synthetic data generation process as we noticed it was correlated with undesirable behavior like hallucinating date cutoffs and prefacing responses with ‘‘As an AI language model...’’. 19 19 19 These filtering methods were also applied to the chosen samples in the 32B preference data. We also use majority voting to improve the quality of answers to our synthetic math questions, that is, preventing SFT on incorrect math answers. For our Persona MATH 20 20 20 Filtered dataset here: https://huggingface.co/datasets/allenai/tulu-3-sft-personas-math-filtered and Grade School Math 21 21 21 Filtered dataset here: https://huggingface.co/datasets/allenai/tulu-3-sft-personas-math-grade-filtered datasets from Tülu 3, we only include prompts and completions where the model reaches a majority vote over 5 completions. In total, this SFT mix contains 866,138 prompts.

##### Preference Finetuning (PreFT) with DPO

The core strategy of the Tülu 3 pipeline for PreFT is building upon and scaling the UltraFeedback pipeline ( Cui et al., 2023 ) for generating synthetic preferences across data for our target domains. We include on-policy data by sampling responses from some development OLMo 2 SFT models at both 7B and 13B, with independent datasets for each.

From Tülu 3, we updated our model pool to only include models with permissible licenses as shown in Table 25 in the Appendix. We made a minor shift from Tülu 3 on the exact prompts used for DPO – we obtain our prompts from several sources listed in Table 27 , resulting in datasets of 366.7k prompts for 7B and 377.7k prompts for 13B. Given this set of prompts, we generate responses from a pool of 20 models of different families and sizes.

To create synthetic preference data we use GPT-4o-2024-08-06 as an LM judge ( Zheng et al., 2023 ) and prompted it to rate completions based on helpfulness, truthfulness, honesty, and instruction-following aspects. We then binarize the ratings across aspects by following Argilla’s method 22 22 22 See https://huggingface.co/datasets/argilla/ultrafeedback-binarized-preferences . : we get the average rating across all aspects, take the highest-rated completion as the chosen response, and sample from the remaining completions for the rejected response.

The 1B and 32B DPO models were trained with the same on-policy methodology.

##### Reinforcement Learning with Verifiable Rewards (RLVR)

RLVR is a novel finetuning technique used to target specific domains where prompts with verifiable answers can be constructed. For example, with a math problem, the RL algorithm Proximal Policy Optimization (PPO) ( Schulman et al., 2017 ) only receives a reward if the answer is correct. For more details, see Lambert et al. (2024) .

Following preference tuning, we trained 7B and 13B reward models using the on-policy 7B and 13B preference dataset. Next, we applied RLVR to the highest-performing 7B and 13B DPO checkpoints with a combined dataset comprising GSM8K, MATH training sets, and prompts with constraints from Lambert et al. (2024) . For RLVR, we initialize PPO’s value function from the corresponding RMs, which is shown to help improve average scores across evaluations ( Lambert et al., 2024 ) . After the initial RLVR training pass on the 13B model, we observe that its performance on GSM8K and MATH was lower than a previous development instruct model. Consequently, we perform two additional RLVR training iterations: first on the GSM8K training set, followed by the MATH training set. The models selected at the end of the RLVR stage constitute the final OLMo 2 Instruct models.

For the 1B and 32B model, we performed RLVR with Group Relative Policy Optimization (GRPO) ( Shao et al., 2024 ) , which forgoes the need for a reward model. The evaluation metrics for this 32B model are shown in Fig. 14 .

OLMo-2-1124-13B-RLVR1 OLMo-2-1124-13B-RLVR2 OLMo-2-1124-13B-Instruct (Final RLVR)

OLMo-2-1124-7B-Instruct

##### Hyperparameter selection

We perform the following hyperparameter tuning for the 7 and 13B models. At each stage we experiment with 1 random seed initially to arrive on a configuration and up to 4 with final hyperparameters. The final hyperparameters are marked with ( \varheartsuit {{\color[rgb]{0.9414,0.3203,0.6133}\varheartsuit}} ): 1. SFT: We sweep over learning rates 1 × 10 − 5 1\times 10^{-5} , 2 × 10 − 5 2\times 10^{-5} ( \varheartsuit {{\color[rgb]{0.9414,0.3203,0.6133}\varheartsuit}} ), 3 × 10 − 5 3\times 10^{-5} for the 7B model and 1 × 10 − 6 1\times 10^{-6} , 4 × 10 − 6 4\times 10^{-6} , 5 × 10 − 6 5\times 10^{-6} ( \varheartsuit {{\color[rgb]{0.9414,0.3203,0.6133}\varheartsuit}} ), 7.5 × 10 − 6 7.5\times 10^{-6} , 8 × 10 − 6 8\times 10^{-6} for the 13B model.

2. DPO: We sweep over learning rates 5 × 10 − 7 5\times 10^{-7} , 6 × 10 − 7 6\times 10^{-7} , 7 × 10 − 7 7\times 10^{-7} , 8 × 10 − 7 8\times 10^{-7} ( \varheartsuit {{\color[rgb]{0.9414,0.3203,0.6133}\varheartsuit}} - 13B), and 1 × 10 − 6 1\times 10^{-6} ( \varheartsuit {{\color[rgb]{0.9414,0.3203,0.6133}\varheartsuit}} - 7B) for both the 7B model and 13B model.

3. RM: We train with 3 × 10 − 6 3\times 10^{-6} learning rate and 1 random seed for the 7B and 13B models, respectively.

4. RLVR: We sweep over beta values 0.03, 0.05, 0.07 ( \varheartsuit {{\color[rgb]{0.9414,0.3203,0.6133}\varheartsuit}} - 7B), and 0.1 ( \varheartsuit {{\color[rgb]{0.9414,0.3203,0.6133}\varheartsuit}} - 13B). For 13B model, we also sweep over learning rates 3 × 10 − 7 3\times 10^{-7} ( \varheartsuit {{\color[rgb]{0.9414,0.3203,0.6133}\varheartsuit}} - 13B), 4 × 10 − 7 4\times 10^{-7} ( \varheartsuit {{\color[rgb]{0.9414,0.3203,0.6133}\varheartsuit}} - 7B). For 13B, we run this sweep on the best model at each RLVR stage.

We conducted a hyperparameter sweep for SFT and DPO, using earlier development checkpoints, with results detailed in Table 17 and Figure 12 . A key finding was that OLMo 2 required significantly higher learning rates compared to the Llama 3.1 training recipe described by Lambert et al. (2024) . Finally, the optimized hyperparameters for our final model are presented in Table 17 and Table 18 .

The post-training for the 32B model occurred after the release of the 7 and 13B models, so the hyperparameter selection proceeded independently. For SFT, we swept over a learning rate of 1 × 10 − 6 1\times 10^{-6} , 2 × 10 − 6 2\times 10^{-6} , 3 × 10 − 6 3\times 10^{-6} , 4 × 10 − 6 4\times 10^{-6} , 5 × 10 − 6 5\times 10^{-6} , with the best performance as 4 × 10 − 6 4\times 10^{-6} where we ran one additional seed to compare performance. For DPO, we swept over learning rates again, from 8 × 10 − 7 8\times 10^{-7} , 1 × 10 − 6 1\times 10^{-6} , 1.5 × 10 − 6 1.5\times 10^{-6} , 2 × 10 − 6 2\times 10^{-6} , 2.5 × 10 − 6 2.5\times 10^{-6} , and the best performance was 2 × 10 − 6 2\times 10^{-6} . For RLVR, the 32B does not need a reward model due to the change to GRPO. Beyond that, the final model was trained with a learning rate of 5 × 10 − 7 5\times 10^{-7} , with a KL beta of 0.1, and 16 samples per prompt.

##### Evaluation of OLMo 2-Instruct

Following Tülu 3 ( Lambert et al., 2024 ) , we evaluate OLMo 2-Instruct on five categories listed in Table 15 . Although Tülu 3 uses six categories including code-related tasks, we exclude this category since code was not a target skill during the development of OLMo 2 . For each of the remaining categories, we use the same evaluations as those used for developing the Tülu 3 recipe. Table 15 also shows the settings and metrics used for each of the evaluations. These match those recommended in Lambert et al. (2024) for the non-code categories.

Table 16 presents the performance of OLMo 2 Instruct variants across different training stages. A comparative analysis of OLMo 2-Instruct ’s performance against similarly-sized open models can be found in Table 7 . Furthermore, Figures 13 and 15 present the training trajectories and key performance metrics for the 13B and 7B models, respectively.

The OLMo 2-Instruct models demonstrate comparable performance to leading open-weight models in the field. Specifically, OLMo 2 13B Instruct achieves results approaching those of Qwen 2.5 14B Instruct while surpassing both Tülu 3 8B and Llama 3.1 8B Instruct in performance benchmarks. The RLVR stage also demonstrated consistent effectiveness across both model scales, leading to notable improvements in evaluation metrics in tandem with increasing the training reward signal.

Finally, we evaluate OLMo 2-Instruct on the unseen evaluation suite from Lambert et al. (2024) without the code evaluation tasks. The Instruct scores on the unseen evaluation suite are shown in Table 24 .

## 6 Deep Dive: Infrastructure as a Research Catalyst

LM training is famously compute intensive. Training large models requires state-of-the-art hardware, and a lot of work goes into making it run efficiently. Gains in efficiency can be translated into higher token counts or more parameters, directly affecting the quality of the final model. GPUs are at the core of this infrastructure, investment in other processes and systems is required to make them perform at peak efficiency. Data centers need high-speed interconnect between compute nodes to make sure expensive GPUs never have to wait for data to arrive. Training jobs need access to large amounts of fast, reliable storage for access to training data. GPUs have higher failure rates than most other hardware, and a single training run might require thousands of them at the same time, making effective monitoring and replacement policies a necessity. This section provides details about our hardware and software investments to support OLMo 2 workloads.

### 6.1 Clusters

OLMo 2 is trained on two Ai2 clusters, Jupiter and Augusta. Despite hardware and architectural differences, both clusters provided sufficient training throughput. Beaker, Ai2’s workload management system, allows researchers to migrate workloads from one cluster to another, and both the 7B and 13B variants were trained partially on both clusters, with the bulk of the 7B training on Jupiter, and the bulk of 13B training on Augusta.

#### 6.1.1 Jupiter

Jupiter is a 128-node GPU cluster located in Austin, Texas. It is operated by Cirrascale Cloud Services 23 23 23 cirrascale.com .

##### Compute

It consists of 1,024 NVIDIA H100 GPUs, each with 80GB HBM3 running at 700W. The GPUs are spread across 128 servers with 2x Intel Xeon Platinum 8468 CPUs, 2 TB of DDR5 system memory, and 18 TB of local NVMe storage.

##### Storage

The servers are connected via a 800 Gbps local network to a WEKA high performance storage cluster 24 24 24 weka.io . This cluster has 1 PB of NVMe SSD storage with 11 physical servers, and 5 PB of HDD storage spread across 12 hosts. The Jupiter GPU servers have two bonded 25 Gbps Mellanox ethernet cards each, providing a total of 50 Gbps of throughput per host. In benchmarks, we reach 761 Gbps of read/write throughput using 64 client machines.

##### Interconnect

Cross-node GPU communication is provided via RDMA over InfiniBand and a 2-Tier Rail Optimized ( Wang et al., 2023a ) , balanced, full-bisected network. Each physical server has eight 400 Gbps InfiniBand cards, providing a maximum total throughput per host of 3200 Gbps. This setup allows Ai2 to run dozens of distributed training workloads simultaneously on the same cluster without topological scheduling.

##### Cooling

The Jupiter servers are racked in Dynamic Density Cabinets 25 25 25 cirrascale.com/products-and-services/cabinet-technologies . Each cabinet includes 5 servers with dedicated cooling and power. Each cabinet is a closed system, circulating air through an overhead compartment where it is cooled via heat transfer to water. This approach allows the datacenter to achieve a power usage efficiency (PUE) of 1.2. Under heavy utilization, our H100 GPUs reach a peak temperature of 75°C; average GPU temperatures are between 60°C and 65°C.

#### 6.1.2 Augusta Cluster

The Augusta cluster is a 160-node GPU cluster provided by Google Cloud. The physical servers are located in Council Bluffs, Iowa.

##### Compute

The cluster is made up of A3 Mega virtual machines 26 26 26 a3-megagpu-8g , more information at cloud.google.com/compute/docs/accelerator-optimized-machines , each with 8 NVIDIA H100 GPUs.

##### Storage

Augusta workloads use Google Cloud Storage for speeds up to 1 GB/s per VM. We ensure portability by abstracting storage interactions into common libraries supporting both file- and object-based APIs.

##### Interconnect

Each GPU has a dedicated Ethernet NIC. Fast cross-node GPU communication is achieved using GPUDirect-TCPXO, gVNIC, and compact node placement. This arrangement takes advantage of Google’s Jupiter data center network technology and the Titanium system with tiered offloading and full-bandwidth reconfigurable optical links. This provides bandwidth similar to non-blocking network fabrics.

##### Cooling

The Augusta servers are air-cooled and the Iowa campus in which they are located reported a trailing twelve-month power usage efficiency (PUE) of 1.12.

### 6.2 Beaker

OLMo 2 workloads were scheduled using Beaker ( Guerquin, 2022 ) , a custom workload management system. Beaker benefited OLMo 2 in two key ways:

##### Portability

Beaker’s architecture can take advantage of GPUs across 3 different data centers with minimal code changes. It can be run anywhere running a single Linux daemon that is packaged as a statically linked binary. Typically, workloads can be moved from one location to another by changing a single line of code.

##### Isolation