2 OLMo 2 Furious — part 4/7
Cleaned-text code points [46977, 62408). Bibliography omitted here; source.md includes it.
• FineWeb Edu classifier from Penedo et al. (2024) . This model 12 12 12 HuggingFaceFW/fineweb-edu-classifier is fine-tuned from the Arctic Embed M 13 13 13 Snowflake/snowflake-arctic-embed-m encoder ( Merrick et al., 2024 ) on over 400,000 web pages 14 14 14 datasets/HuggingFaceFW/fineweb-edu-llama3-annotations labeled by Llama 3 70B Instruct. This classifier scores documents from 0 to 5 according to adherence to academic topics and polished content.
Following Li et al. (2024) , we use the DCLM FastText classifier with a threshold of 0.03311014 0.03311014 , which retains approximately 65.6% of the web subset. We combine this filter with the scores from FineWeb Edu classifier; we experiment by retaining documents with score over 3 (5.8% retained), as well as a more relaxed threshold of 2 (20.3% retained).
##### Instruction data and Q&A pairs
We leverage the same subset of FLAN Wei et al. (2021) ; Longpre et al. (2023) from Dolma 1.7 ( Soldaini et al., 2024 ) . We decontaminated this source by extracting training, validation, and test instances from all tasks in our evaluation suite (Section § 2.5 ) and removed FLAN documents with 10% or more overlapping ngrams with any task instance.
We source question and answer pairs from the Stack Exchange network, a collection of 186 forums dedicated to a wide variety of topics. Content on Stack Exchange network is licensed under various commercial-friendly Creative Common licenses. We use the latest database dump (September 30 th \text{30}^{\text{th}} , 2024) at the time of writing, which is distributed by the Internet Archive 15 15 15 archive.org/details/stackexchange_20240930 . We filter questions to those that have an accepted answer; further, we remove Q&A pairs whose questions have fewer than 3 votes or answers have fewer than 5 votes. Once filtered, we concatenate questions and answers together using a sequence of new lines that contains one more \n than the longest sequence of newlines in either the question or answer.
##### Code
We evaluate retaining the same subset of code used during pretraining; furthermore, we consider smaller, curated sources of code interleaved with natural supervision, such as docstrings in CodeSearchNet ( Husain et al., 2019 ) ; Q&A pairs from StackExchange described in the paragraph above also contain code.
##### Academic, encyclopedic and other reference content
We source high-quality non-web datasets from Dolma 1.7 ( Soldaini et al., 2024 ) . This includes peS2o ( Soldaini and Lo, 2023 ) , Wikipedia, and Wikibooks, Gutenberg books, arXiv and StackExchange (from Red-Pajama v1; Together AI, 2023 ), Algebraic Stack (ProofPile II; Azerbayev et al., 2023 ).
##### Math
In parallel to developing the math subset of Dolmino Mix 1124 (Section § 4.4 ), we consider preliminary math subset to gauge how math documents combine with the non-math portion of the mix. In particular, we used OpenWebMath ( Paster et al., 2023 ) , the train split of GSM8K ( Cobbe et al., 2021 ) , the train split of the permissively licensed (“commercial”) subset of MathPile ( Wang et al., 2023b ) , and AutoMathText ( Zhang et al., 2024b ) .
Results of mixes shown in Table 10 are summarized in Table 11 . All results correspond to mid-training runs on 50 billion tokens, initialized from a 7B model checkpoint pretrained on 4 trillion tokens.
We find that, as noted in Section § 4.1 , learning rate anneal ( \sans PT Mix) alone yields notable improvements across all averages (OLMES + 4.4 +4.4 ; OLMES-Gen + 1.3 +1.3 ; MMLU + 20 +20 ), but not on our math development set (GSM* − 1.5 -1.5 ). Switching to mixes that contain higher quality web data and reference content further improves performance: \sans Web \sans FW \sans 2 \sansFT \sans7 {}^{\text{{\sans{FT}}}_{\text{{\sans{7}}}}}_{\text{{\sans{FW}}}_{\text{{\sans{2}}}}} further improves + 1.2 +1.2 points over \sans PT Mix in OLMES and + 1.3 +1.3 in MMLU; it is slightly worse on OLMES-Gen ( − 0.4 -0.4 ) and within margin of error on GSM* ( + 1.5 +1.5 ). Finally including instruction data and math sources in the mix yields the best performance. \sans Web \sans FW \sans 2 \sansFT \sans7 {}^{\text{{\sans{FT}}}_{\text{{\sans{7}}}}}_{\text{{\sans{FW}}}_{\text{{\sans{2}}}}} \sans + Math + Ins mix achieves best overall results, with + 1.7 +1.7 on OLMES, + 5.7 +5.7 on generative tasks, + 1.3 +1.3 on MMLU, and + 19.5 +19.5 on GSM*. We note that \sans Web \sans FW \sans 2 \sansFT \sans7 {}^{\text{{\sans{FT}}}_{\text{{\sans{7}}}}}_{\text{{\sans{FW}}}_{\text{{\sans{2}}}}} \sans + Math mix performs slightly better on math tasks, motivating our investigation in better math subsets that combine well with other high-quality sources in Section § 4.4 .
### 4.4 Dolmino Mix 1124 : Math Mix
Early mid-training mixes ( \sans Web ∗ ∗ \text{\sans{Web}}_{\ast}^{\ast} only rows in Table 11 ) show models struggle in math-related benchmarks. Thus, improving performance on these sets is a central focus of our mid-training investigations. We investigate both human-authored and synthetically generated or augmented data; we derived the latter through an iterative procedure aimed at fixing common errors in our math validation sets.
We describe both the data sources and their generation/filtration procedure in Section § 4.4.1 ; then, in Section § 4.4.2 , we detail microanneals , the experimentation technique we use to finalize math sources. The resulting mix is summarized in Table 5 .
#### 4.4.1 Math Sources
##### TuluMath
We follow the recent persona-driven methodology in Chan et al. (2024) to generate math synthetic data. The key idea is to use different personas (e.g., “A machine learning researcher focused on neural networks”) with a data synthesis prompt (e.g., “create a math problem”) to steer an LM to synthesize data with corresponding perspectives. Specifically, we condition on available personas from Persona Hub ( Chan et al., 2024 ) to generate prompts targeting Math problems both those that require advanced mathematical skills as well as grade school problems. We zero-shot-prompt GPT-4o 16 16 16 2024-08-06 to generate problems that are unique and specific to a given persona input. Having generated the problems, we then generate multi-step math solutions using GPT-4o . Exact prompts used to generate problems and solutions are provided in Appendix Figures 24 and 25 . In total, we collected ∼ \sim 230M synthetic math tokens.
##### DolminoSynthMath
This is a collection of 28M synthetic math tokens designed specifically to improve performance on GSM8K as well as raw mathematical calculations. It is composed of three parts: first we generate 11M tokens of basic mathematical question and answer pairs such as “77 * 14 = 1078” and pair each of these with a variety of prompts. We find that including such data dramatically mitigates the mistakes our model makes within individual CoT reasoning steps at inference time. Next we include a custom collection of 7,924 synthetic GSM8K examples, which are produced by consuming a GSM8K training example and replacing all of its numbers in both the provided question and answer, with the hope that this would provide signal to the model to extract the computation graph from a word problem and ignore irrelevant semantic features. Finally we include a MIND-rewriting ( Akter et al., 2024 ) of each of the GSM8K training examples, where the synthetic data was generated using Qwen2.5-7B-Instruct ( Qwen et al., 2024 ) .
##### TinyGSM-MIND
We generated approximately 6.5B tokens of synthetic math data from rewritten versions of Tiny-GSM ( Liu et al., 2023a ) . Tiny-GSM is a collection of 11M synthetic GSM8K-like questions, where the answers are provided in the form of python code. We filter this set to only include answers that have code that is executable and only contains statements that are variable assignments. We then annotate each line of the code that is an assignment operator with the numerical value of the resulting variable. Then we pass all of these annotated examples to Qwen2.5-7B-Instruct to be rewritten in the style of MIND ( Akter et al., 2024 ) using the ‘Two Students’ and ‘Problem Solving’ prompts.
##### MathCoder2-Synthetic
We emulate the synthetic data generation procedure of MathCoder2 ( Lu et al., 2024 ) to filter existing synthetic data from open-source repositories. In particular, we collect the synthetic textbooks from HuggingFace user Ajibawa-2023 , 17 17 17 datasets/ajibawa-2023/Maths-College , {}^{\text{,}} 18 18 18 datasets/ajibawa-2023/Education-College-Students and from the M-A-P Matrix dataset and perform additional filtering on them. In particular we train a FastText classifier as follows: we ask GPT-4o to annotate 10,000 OpenWebMath examples ( Paster et al., 2023 ) as either math-related or non-math-related; we then use these as positive and negative examples for a FastText classifiers. We apply this classifier to the synthetic textbooks and only keep the math-related ones.
##### ProofPile OWM-Filtered
We use the same OpenWebMath filter generated in the previous step and apply it to Metamath ( Yu et al., 2023 ) and CodeSearchNet ( Husain et al., 2019 ) .
##### GSM8K-Train
Finally, we include the training split of GSM8K ( Cobbe et al., 2021 ) .
#### 4.4.2 Evaluating Math Data with Microanneals
To select the highest quality subset of all available and synthetic math data, we perform a series of several microanneals , which were annealing runs focused on small math subsets. The general recipe for these microanneals is as follows: 1. identify a source or small collection of math sources that we want to assess the data quality of;
2. collect roughly the same quantity of data from the general data mix (e.g., DCLM) as from the math sources to ensure a mixture of high-quality web text alongside domain-specific math;
3. train this 50/50 mixture as if it were an annealing run, making sure to linearly drive the learning rate down at the proper rate for this smaller collection of data.
This procedure facilitates evaluating the quality of individual data sources at a fraction of the cost of a full annealing run. In total, we run 19 separate microanneals with a total token count of 130B tokens, equivalent to less than 3 full 50B annealing runs. Putting this cost into perspective, the totality of the 19 microanneals requires less compute than the 3 50B token souping ingredients used for our 7B model. More explicitly, it shows improvements at a much finer-grained data-source resolution, with results visible after training for less than 10B tokens.
We illustrate how microanneals lead to our final math mix through three sets of experiments reported in Table 12 . The primary evaluation metrics we use to evaluate the quality here is MMLU, and GSM*, which is our 200-example subset of the GSM8K evaluation set. Note that one goal of mid-training is to improve GSM8K performance, but we only allow ourselves to inspect performance on 200 of the 1319 GSM8K examples to inform decisions about data mixtures.
##### Microanneal experiment 1: domain specific data is helpful even in small proportions
We run the following experiment: starting from a 7B model that has completed pretraining, and a mixture of TuluMath, DolminoSynthMath, Metamath, CodeSearchNet, and GSM8K-Train, accounting for approximately 200M tokens, we train on both a 35/65 math/DCLM mixture and a 10/90 mixture and evaluate both the MMLU and GSM*. We see that the pre-anneal had a GSM* score of 28.5, the 35/65 mixture yields a GSM* of 63.5, and the 10/90 mixture yields a GSM* of 61. This suggests that it is not strictly necessary to have a large proportion of domain-specific data in the annealing mixture, just that domain-specific data is present.
##### Microanneal experiment 2: some duplication is beneficial
Starting from the same setup as the previous experiment, we duplicate the math data for a total of two copies, and four copies. We see that one copy of the math yields a GSM* score of 61, two copies yields a score of 66, and four copies yields a score of 65. This suggests that even if there is a scarcity of high-quality domain-specific data, duplicating it a small number of times can still provide some gains.
##### Microanneal experiment 3: rewriting can help dramatically
Here we once again start with a 7B model that has completed pretraining and evaluate the effect that rewriting Tiny-GSM into a natural language format has on GSM* evaluation scores. Recall that Tiny-GSM has answers written in the form of code, and that our pretraining mix is only 2% code. We run a microannealing run on a mixture using an inline-annotated form of TinyGSM and compare it to just the ‘Problem Solving’ MIND rewritten variant of TinyGSM. Relative to the baseline, the code version of TinyGSM degrades GSM* performance, while the rewritten version dramatically improves the performance. This suggests the power of rewriting as a tool to cheaply convert data to a more amenable form for training.
### 4.5 Final Midtraining mix and Checkpoint Soups
Source Tokens 50B 100B 300B Source % Mix % Source % Mix % Source % Mix % Filtered DCLM 752B 3.23 47.2 6.85 50.2 20.78 51.9 Decontam. FLAN 17.0B 50.0 16.6 100 16.7 200 11.3 StackExchange Q&A 1.26B 100 2.45 200 2.47 400 1.68 peS2o 58.6B 5.15 5.85 16.7 9.52 100 19.4 Wikipedia/Wikibooks 3.7B 100 7.11 100 3.57 400 4.86 Dolmino Math 10.7B 100 20.8 200 17.5 400 10.8
The final composition of Dolmino Mix 1124 is shown in Table 5 . As previously mentioned, we sample 3 mixes of 50B, 100B, and 300B tokens; composition of each is summarized in Table 13 . Since experiments in Section § 4.3 and § 4.4.2 show that keeping mixing proportion roughly constant across sources is beneficial, we repeat Stack Exchange Q&A data and mid-training math data twice for the 100B tokens mix, and four times for the 300B mix; additionally, we repeat FLAN twice and Wiki data four times for the 300B mix. Across all mixes, filtered web data from the DCLM baseline represents roughly 50% of the total tokens budget.
We train OLMo 2 7B on the 50B mix. To account for the larger batch size (Section § 2.3 ), we use the 100B mix for OLMo 2 13B, ensuring the same number of steps during learning rate anneal. Further, we experiment with a longer anneal phase with OLMo 2 13B using the 300B mix. We follow the same procedure for the 32B model.
##### Mid-training model merging or “soups”
Performing a naïve average of multiple model checkpoints trained with a different data order has been proven effective in both computer vision ( Wortsman et al., 2022 ) and language modeling ( Li et al., 2024 ) applications. We confirm the effectiveness of this approach, also known as model merging or “souping” , on six different mid-training mixes, as shown in Table 14 . For all experiments, we find that merging 3 checkpoints annealed on three permutations of the same data mix consistently produces equal or better performance than any individual training run.
Based on this evidence, we extensively use model merging to obtain our final OLMo 2 7B and 13B models. For OLMo 2 7B, we average three checkpoints trained on the 50B sample of Dolmino Mix 1124 . For OLMo 2 13B and 32B, we average four checkpoints: three trained on the 100B sample, and one trained on a 300B sample; we find this approach to be empirically better than averaging just the three 100B runs alone.
## 5 Deep Dive: Post-training Pipeline