2 OLMo 2 Furious — part 1/7

Cleaned-text code points [0, 15735). Bibliography omitted here; source.md includes it.

\authorOne [ ]OLMo Team \authorTwo [ 1 ]Pete Walsh \authorTwo [ 1 ]Luca Soldaini \authorTwo [ 1 ]Dirk Groeneveld \authorTwo [ 1 ]Kyle Lo \authorThree [ 1 ]Shane Arora \authorThree [ 1 ]Akshita Bhagia \authorThree [ 1 ]Yuling Gu \authorThree [ 1 ]Shengyi Huang \authorThree [ 1 ]Matt Jordan \authorThree [ 1 ]Nathan Lambert \authorThree [ 1 ]Dustin Schwenk \authorThree [ 1 ]Oyvind Tafjord \authorFour [ 1 ]Taira Anderson \authorFour [ 1 ]David Atkinson \authorFour [ 1 ]Faeze Brahman \authorFour [ 1 ]Christopher Clark \authorFour [ 1 ]Pradeep Dasigi \authorFour [ 1 ]Nouha Dziri \authorFour [ 1 ]Allyson Ettinger \authorFour [ 1 ]Michal Guerquin \authorFour [ 1 ]David Heineman \authorFour [ 1 , 2 ]Hamish Ivison \authorFour [ 1 , 2 ]Pang Wei Koh \authorFour [ 1 , 2 ]Jiacheng Liu \authorFour [ 1 ]Saumya Malik \authorFour [ 1 , 3 ]William Merrill \authorFour [ 1 ]Lester James V. Miranda \authorFour [ 1 ]Jacob Morrison \authorFour [ 1 ]Tyler Murray \authorFour [ 1 ]Crystal Nam \authorFour [ 1 ]Jake Poznanski \authorFour [ 1 , 2 ]Valentina Pyatkin \authorFour [ 1 ]Aman Rangapur \authorFour [ 1 ]Michael Schmitz \authorFour [ 1 ]Sam Skjonsberg \authorFour [ 1 ]David Wadden \authorFour [ 1 ]Christopher Wilhelm \authorFour [ 1 ]Michael Wilson \authorFour [ 2 ]Luke Zettlemoyer \authorFive [ 1 , 2 ]Ali Farhadi \authorFive [ 1 , 2 ]Noah A. Smith \authorFive [ 1 , 2 ]Hannaneh Hajishirzi 1 ]Allen Institute for AI 2 ]University of Washington 3 ]New York University \contribution [] OLMo 2 was a team effort. marks core contributors. See full author contributions here . \metadata [ OLMo 2 Base:] OLMo-2-1124-7B OLMo-2-1124-13B OLMo-2-0325-32B \metadata [ OLMo 2 Instruct:] 7B-Instruct 13B-Instruct 32B-Instruct \metadata [

Base Data:] olmo-mix-1124 \sans (pretrain) dolmino-mix-1124 \sans (midtrain) \metadata [ Instruct Data:] SFT: 7B , 13B , 32B DPO: 7B , 13B , 32B RLVR \metadata [

Training Code:] OLMo \sans (pretrain v1) OLMo-core \sans (pretrain v2) open-instruct \sans (posttrain) \metadata [ Eval & Data Code:] olmes \sans (eval suite) dolma \sans (data curation) \metadata [

Training Logs:] 7B 13B 32B \metadata [

Demo:] playground.allenai.org \metadata [

Contact:] olmo@allenai.org

# 2 OLMo 2 Furious

###### Abstract

We present OLMo 2 , the next generation of our fully open language models. OLMo 2 includes a family of dense autoregressive language models at 7B, 13B and 32B scales with fully released artifacts—model weights, full training data, training code and recipes, training logs and thousands of intermediate checkpoints. In this work, we describe our modified model architecture and training recipe, focusing on techniques for achieving better training stability and improved per-token efficiency. Our updated pretraining data mixture introduces a new, specialized data mix called Dolmino Mix 1124 , which significantly improves model capabilities across many downstream task benchmarks when introduced via late-stage curriculum training (i.e. specialized data during the annealing phase of pretraining). Finally, we incorporate best practices from Tülu 3 to develop OLMo 2-Instruct , focusing on permissive data and extending our final-stage reinforcement learning with verifiable rewards (RLVR). Our OLMo 2 base models sit at the Pareto frontier of performance to training compute, often matching or outperforming open-weight only models like Llama 3.1, Qwen 2.5, and Gemma 2 while using fewer FLOPs and with fully transparent training data, code, and recipe. Our fully open OLMo 2-Instruct models are competitive with open-weight only models of comparable size and even some proprietary models like GPT-3.5 Turbo and GPT 4o Mini.

## 1 Introduction

The open language model ecosystem has grown rapidly in the past year. We’ve seen a surge in open weights models from established developers—Llama 3 ( Grattafiori et al., 2024 ) , DBRX ( Databricks, 2024 ) , Yi 1.5 ( Young et al., 2024 ) , Qwen 2 ( Yang et al., 2024a ) , Falcon ( TII, 2024a ; TII, 2024b ) , Mistral ( Mistral, 2024a ) , Ministral ( Mistral, 2024b ) , Phi ( Abdin et al., 2024a ; Abdin et al., 2024b ) — and new contributors— Gemma ( Gemma Team et al., 2024a ; Gemma Team et al., 2024b ; Team et al., 2025 ) , Grok ( X.AI, 2023 ) , Command R ( Cohere, 2024a ; Cohere, 2024c ; Cohere, 2024b ) —substantially closing the gap between publicly available and closed systems ( Cottier et al., 2024 ) . Yet, these open-weights models are only the final artifacts of sophisticated language model recipes and complex development pipelines, and by themselves are not sufficient to support diverse forms of research into language model behaviors and uses.

In response, prior works including our first OLMo ( Groeneveld et al., 2024 ) , Pythia ( Biderman et al., 2023 ) , Amber ( Liu et al., 2023c ) , DCLM ( Li et al., 2024 ) , MAP Neo ( Zhang et al., 2024a ) and SmolLM ( Allal et al., 2024a ; Allal et al., 2024b ) have adopted a fully open approach , releasing not just model weights but also training data, training code and well-documented recipes to support reproduction. Artifacts from fully open language modeling efforts have played a crucial role in studying training dynamics ( Land and Bartolo, 2024 ; Jin and Ren, 2024 ) , concept acquisition ( Chang et al., 2024 ) , and memorization ( Antoniades et al., 2024 ; Shaib et al., 2024 ) in language models. Despite these developments, a gap remains between the models with the best reported performance and that of open models.

Modern language model development is an iterative process, whereby limitations of current iterations motivate future development. Our previous release ( Ai2, 2024 , OLMo-0424 ;) focused on improving performance on key tasks ( e.g. , MMLU) through better pretraining data mixing and curricula. In this technical report, we introduce OLMo 2 , a new family of 7B, 13B and 32B models trained on up to 6T tokens. On English academic benchmarks, these models are competitive with the open weight Llama 3.1, Qwen 2.5, and Gemma 2 families of models (Figure 1 ). We further validate our pretrained model is an effective base model for downstream post-training by applying our Tülu 3 recipe ( Lambert et al., 2024 ) . The resulting family of models, called OLMo 2-Instruct , are competitive with powerful open-weights only models and even some popular proprietary models like GPT-3.5 Turbo and GPT 4o Mini. This technical report focuses on four key areas we targeted during development of OLMo 2 : • Pretraining Stability. Language model training runs are often plagued by training instabilities and loss spikes, which are costly and known to be a detriment to final model performance. We discuss techniques we used to improve training stability, which was critical to ensuring performance of the final trained model (Section § 3 ).

• Mid-training Recipe. OLMo-0424 ( Ai2, 2024 ) , DBRX ( Databricks, 2024 ) , and Llama 3 ( Grattafiori et al., 2024 ) demonstrated the usefulness of data curricula for pretraining, as discussed by Blakeney et al. (2024) . We discuss the advantages of splitting pretraining into two stages, with the latter mid-training stage being used to infuse new knowledge and patch deficiencies in capabilities. Further, we show how data sources for mid-training can be independently assessed to reduce experimentation cost through a technique we call micro-annealing (Section § 4 ).

• Post-training Pipeline. A key deliverable for a successful base model is its ability to be finetuned to downstream use-cases. We introduce OLMo 2-Instruct built on the Tülu 3 recipe ( Lambert et al., 2024 ) , and show how improvements in base models translated to better chat variants. We focus on permissive data and expand the reinforcement learning with verifiable rewards (RLVR) pipeline to multiple stages for maximum performance (Section § 5 ).

• Infrastructure as a Research Catalyst. High performance and reliable infrastructure is crucial for successful pretraining; yet, many pretraining papers do not discuss their training stack, or gloss over crucial details. We discuss changes from OLMo-0424 that enable the improvements of OLMo 2 , and how investing in solutions that let us monitor and orchestrate infrastructure helped us reduce failure rates and increase cluster utilization (Section § 6 ).

Alongside these deep dives, we provide a description of the full model development procedure in Section § 2 : training data, pretraining, post-training, and evaluation. We highlight changes from OLMo 1 and OLMo-0424 when appropriate, and reference related projects, such as our scaling laws effort to efficiently estimate model downstream performance ( Bhagia et al., 2024 ) and benchmark standardization through the OLMES evaluation framework ( Gu et al., 2024 ) .

## 2 OLMo 2 Family

This section provides an overview of OLMo 2 and highlights improvements over OLMo-0424 and previous OLMo models 1 1 1 Model architecture changes over OLMo 1 and OLMo-0424 are described in Section § 2.1 ; for an overview of data and training recipes, see Groeneveld et al. (2024) and Ai2 (2024) respectively. . The OLMo 2 family has more tokens, more parameters, and has better downstream task results compared to OLMo-0424 . We explain the crucial details required to achieve competitive results in our mission of making state-of-the-art language models accessible. Accordingly, we release all training code, data, and recipes openly under the Apache 2.0 license wherever possible, and under the most permissive available license otherwise.

### 2.1 Model Architecture

Table 1 provides an overview of how the model architecture has evolved through iterations in the OLMo family. We provide details below:

We adopt a decoder-only transformer architecture based on Vaswani et al. (2017) , and deliver 7B, 13B and 32B parameter variants as described in Table 3 . Our architecture is very similar to the first iteration of OLMo ( Groeneveld et al., 2024 ) , with several changes to improve training stability (see Section § 3 ) and performance. The original OLMo modified the decoder-only transformer architecture ( Vaswani et al., 2017 ) with: • No biases: We exclude all bias terms from our architecture ( Groeneveld et al., 2024 ; Chowdhery et al., 2022 , inter alia ) .

• SwiGLU activation function: We use the SwiGLU activation function ( Shazeer, 2020 ) and set the corresponding hidden size to approximately 8 3 ​ d \frac{8}{3}d , but increased to the closest multiple of 128 ( 11,008 11,008 for our 7B model) to improve throughput.

• Rotary positional embeddings (RoPE): We replace absolute positional embeddings with rotary positional embeddings (RoPE; Su et al., 2021 ).

When building OLMo-0424 , we made modifications for training stability and downstream performance: • QKV Clipping: For training stability, also as seen in DBRX ( Databricks, 2024 ) .

• Increased context: From 2048 to 4096.

Finally, this work introduces OLMo 2 which made further modifications:

• RMSNorm: We use the RMSNorm ( Zhang and Sennrich, 2019 ) variant of LayerNorm ( Ba et al., 2016 ) without a bias term to normalize activations, instead of nonparametric LayerNorm.

• Reordered norm: We normalize the outputs to the attention and feedforward (MLP) layers within each transformer block, instead of the inputs. So the formula for each block becomes: 𝒉 := 𝒙 + RMSNorm ​ ( Attention ​ ( 𝒙 ) ) \displaystyle\boldsymbol{h}:=\boldsymbol{x}+\text{RMSNorm}(\text{Attention}(\boldsymbol{x})) (1) 𝒉 out := 𝒉 + RMSNorm ​ ( MLP ​ ( 𝒙 ) ) \displaystyle\boldsymbol{h}_{\text{out}}:=\boldsymbol{h}+\text{RMSNorm}(\text{MLP}(\boldsymbol{x})) (2) where 𝒙 \boldsymbol{x} is the input to the layer, 𝒉 \boldsymbol{h} is an intermediate hidden state, and 𝒉 out \boldsymbol{h}_{\text{out}} is the output. This strategy was first proposed by Liu et al. (2021) to stabilize training.

• QK-norm: Following Dehghani et al. (2023b) we normalize the key and query projections with RMSNorm before calculating attention. This avoids attention logits being too large, which can lead to training loss divergence.

• Z-Loss: Following Chowdhery et al. (2022) , Chameleon Team (2024) , and Wortsman et al. (2023) , we adopt z-loss regularization, as it has been empirically shown to improve run stability.

• RoPE θ = 𝟓 ​ e ​ 𝟓 \boldsymbol{\theta=5e5} : We increase the RoPE θ \theta to 500,000 from 10,000. This approach increases the resolution of positional encoding, matching Grattafiori et al. (2024) .

### 2.2 Tokenizer

OLMo 1 and OLMo-0424 were trained using a modified version of the GPT-NeoX-20B tokenizer ( Black et al., 2022 ) that includes special tokens |||PHONE_NUMBER||| , |||EMAIL_ADDRESS||| , and |||IP_ADDRESS||| , which were used to mask personal identifiable information.

As suggested by Tao et al. (2024) , we employ a larger tokenizer vocabulary for OLMo 2 . We borrow pre-tokenizer and vocabulary from cl100k , the tokenizer developed for GPT-3.5 ( OpenAI, 2023a ) and GPT-4 ( OpenAI, 2023b ) , which is licensed under Apache 2.0 2 2 2 github.com/openai/tiktoken/issues/92 . To maintain backwards compatibility with early Dolma data sources, we add the same masking tokens used in previous OLMo models. 3 3 3 Specifically, these tokens such as |||IP_ADDRESS||| appear in early subsets of Dolma dataset. We opt to keep them in vocabulary so that, if tokenizing any of these older sources, they will not get split into multiple tokens.

We compare the two tokenizers at a smaller scale in Table 2 . We see measurable gains when switching to the new tokenizer, particularly in OLMES tasks. Per Tao et al. (2024) , at this model size and compute budget, the larger OLMo 2 tokenizer is at a slight disadvantage; we expect improvement coming from larger vocabulary to be more decisive at larger scales and for models trained on more tokens.

### 2.3 Base Model Training Recipe

Following previous OLMo models, as well as recent advances in curriculum learning ( Blakeney et al., 2024 ; Ibrahim et al., 2024 ) , base OLMo 2 models are trained in two stages each with its corresponding data mix. The first pretraining stage is the longest ( ⩾ 90 % \geqslant 90\% training FLOPs), and uses mostly web-sourced data. In this stage, we use an iteration on our pretraining mix of high-quality web data drawing on other recent open data releases. During the second stage, which we refer to as mid-training ( 5 ​ – ​ 10 % 5\textup{--}10\% of training FLOPs), we up-sample the highest-quality web documents and curated non-web sources; we also employ synthetic data crafted to patch math capabilities of the model.

##### Stage 1: Pretraining

The first stage— pretraining —is the longest ( 90 ​ – ​ 95 % 90\textup{--}95\% of training FLOPs). We report key architecture and training details in Table 3 . Key details include our switch from multi-head attention (MHA) to grouped query attention (GQA) ( Ainslie et al., 2023 ) to scale the 32B model, inspired by its use in concurrent work Qwen 3 ( Yang et al., 2025 ) . OLMo 2 training used random initialization from a truncated normal distribution with a mean of 0 and a standard deviation of 0.02 and a learning rate schedule that warms up the learning rate from 0 to the peak learning rate over 2000 steps, followed by a cosine decay calibrated to reach 10% of the peak learning rate after a specified max tokens.

##### Stage 2: Mid-training

We refer to the shorter second stage as mid-training ( 5 ​ – ​ 10 % 5\textup{--}10\% of training FLOPs), where we linearly decay the learning rate to zero over the remaining length of the run. 4 4 4 While the concept of multiple stages of self-supervised training is not new ( e.g. , Gururangan et al. 2020 ), we adopt the term mid-training from Abdin et al. (2024a) and OpenAI (2024) .