Olmo 3 — part 16/19
Cleaned-text code points [234468, 250350). Bibliography omitted here; source.md includes it.
We support code execution in Python, C++, Java, JavaScript, PHP, Rust and Shell using AWS Lambda functions to grade instances in parallel, isolated environments of up to 50K generations simultaneously. In total, our environments graded 17.2 million generated code samples during Olmo 3 development, with up to 1.5K simultaneously. To ensure reproducibility, we release a lightweight Docker library for code execution without AWS infrastructure 67 67 67 Our code execution environments are publicly available at github.com/allenai/olmes-docker . .
Additionally, OLMo 2 only tracked math and code capabilities after mid-training, as small models exhibit random-chance pass@1 performance on math and code tasks ( Wei et al., 2022 ) . Our base easy suite tracks perplexity over human-written math and code solutions ( Huang et al., 2024b ) , which allows us to broadens the scope of capabilities we track during pre-training.
#### A.4.1 Base Evaluation suites
Using the analysis tools described in the previous section, we construct two evaluation suite for decision making in pre-training: the Base Easy suite for small-scale data decisions and the Base Main suite for in-loop evaluation and mid-training data decisions. We kept the number of in-context examples and generation arguments consistent within each family of tasks, when possible. 68 68 68 We perform all evaluation using vLLM. To prevent performance discrepancies between versions, we pin to v0.9.0.1 for evaluation during development, and pin to v0.11.0 for all evaluation in the final report.
Table 46 describes the task configuration and metrics for the Olmo 3 Base Main evaluation suite. Table 45 provides an overview of the Base Easy suite.
##### Base Easy suite
For multiple-choice BPB, we simply use the correct answer as the continuation. For math BPB, we use the provided human-written solutions from Minerva MATH ( Lewkowycz et al., 2022 ) . For code BPB, we use the gold ‘canontical’ solution as provided in HumanEval and MBPP ( Chen et al., 2021 ; Austin et al., 2021 ) . For BPB over non-Python coding tasks, MultiPL-E did not release gold solutions ( Cassano et al., 2022 ) , so we generate silver continuations for 16 languages using o4-mini-medium 69 69 69 We release this generation set at huggingface.co/datasets/allenai/multilingual_mbpp . Figure 30 shows the scaling behavior of the three base easy task clusters, where we see signal even at very small (190M parameter) model sizes.
One important property of the Base eval suite is that a ranking of two small models on the base easy suite agrees with their ranking on the downstream base main suite. We validate this by measuring rank correlation between the easy and main task suites, as pictured in Figure 31 .
##### Base Main suite
As a result of the clustering procedure, the base main suite tracks 6 task groups: MCQA STEM, MCQA Non-STEM, Gen, Math, Code, Code FIM. Unlike OLMo 2 , we are tracking generative math and code tasks at pre-training. We chose to evaluate pass@k with the largest number of samples such that each task could evaluate on OLMo 2 7B on 1 H100 in under 30 minutes, in order to ensure the eval speed is not bottlenecked by any particular task. For tasks with a large enough n n , we set k = 16 k=16 to match the GRPO group size, which we observed to act as an empirical upper-bound on the possible improvement from RL training. To decide on the the temperature and top-p, we ran a sweep and evaluated 5 models ( OLMo 2 7B, 13B; Qwen 2.5 7B, 13B; Qwen 3 8B; Qwen et al., 2024 ; Yang et al., 2025a ) to find an adequate configuration setting for high scores on both pass@1 and pass@k. Results are shown in Figure 32 , and we select temperature and top-p of 0.6 for all base math and code evaluation.
##### Base Chat suite
During mid-training, we refashion the Chat eval suite (§ 4.1 ) for use evaluating base models, which served as a reference as to whether we expect our model to perform well after the adaptation pipeline. To do this, we used a standard, simple chat template ( Question: {text}\nAnswer: ) across all base models (both Olmo 3 and baseline models) and we included stop tokens to prevent degenerate responses. We also excluded tasks which required an API-based judge (AlpacaEval, SimpleQA) due to cost. In practice, we noticed most of the disagreements between the base main and base chat evaluation suites were due to noise, so we primarily used the base suite for making decisions.
##### Base Long-Context suite
During the long-context extension phase, we evaluate long-context capability using RULER ( Hsieh et al., 2024 ) as our primary development signal. As a complementary held-out set, we also use HELMET ( Yen et al., 2025 ) , noting that the HELMET Recall task directly implements several RULER evaluations (specifically, ruler-niah-mk-2, ruler-niah-mk-3, and ruler-niah-mv). Because we evaluate only base models at this stage, we disable chat templates within HELMET to ensure consistent scoring across models. For HELMET tasks requiring an LLM-as-a-judge, we use its default judge configuration (gpt-4o-2024-05-13). Taken together, RULER guides most model-selection decisions during long-context development, with HELMET providing an additional check on generalization.
##### Base Held-out Suite
We targeted one held-out evaluation task to match each family of capability: MMLU Pro for QA ( Wang et al., 2024a ) , LBPP for code ( Matton et al., 2024 ) , Deepmind Math for math ( Saxton et al., 2019 ) , and BigBench Hard to measure broad coverage across unseen task types ( Suzgun et al., 2022 ) .
#### A.4.2 New Evaluation Benchmarks
##### Basic Skills
We developed a new benchmark, BasicSkills , to measure whether core capabilities are being acquired during pretraining. BasicSkills consists of 6 subtasks: basic arithmetic, string manipulation, simple coding, elementary logical reasoning, basic common sense, and simple pattern recognition. Each task isolates a single skill using a self-contained context that requires no external knowledge or additional information and can be completed through natural text continuation without relying on instruction-following abilities.
##### Gen2MC
One takeaway from OLMo 2 development was a sensitivity to task format. The clustering procedure furhter confirmed this, finding that generative scores rank models similarly as rank choice (RC) QA tasks, disagreeing with ranking of single-token multiple choice (MC) QA tasks (see Figure 5 ). In particular, the short-form generative QA tasks (GenQA in Table 46 ) evaluate by comparing a generated answer to a bank of plausible answers, but these answer banks are often not complete, leading to false negatives. To address this, we introduce the Gen2MC benchmarks, which were constructed by taking the original question/answer pairs and generating incorrect multiple-choice distractor answers using a strong LLM. For each set of generated distractors, we manually review a set of 200 sample questions from the validation set before generating the full dataset. We create Gen2MC tasks for DROP, Jeopardy, NaturalQs, SQuAD, CoQA using GPT-4o for generating distractors, and fall-back to GPT-4.1 in cases where output parsing failed.
##### Masked perplexity
We want our model to perform well on the diversity of requests from real user chat data; however, we don’t want to overfit to the “style” of chat outputs. To avoid this, we use a simple token masking strategy, inspired by work in loss masking ( Mindermann et al., 2022 ) :
1. Fine-tune a 1B model on a tiny subset of the dataset (5̃%) with a small learning rate. The key idea is that we ‘warm up’ to the format of the target set without learning a lot of new knowledge.
2. Compute the token losses of the base model and the fine-tuned model on every sequence in the dataset and compute the difference: log p SFT ( y | x ) − log p base ( y | x ) \log p_{\text{SFT}}(y|x)-\log p_{\text{base}}(y|x)
3. Mask tokens where the difference is greater than some threshold (found by inspection)
4. Also mask the user responses and tool calls (we don’t want to model these for data selection) Use the loss at all the non-masked tokens positions for perplexity evaluations
In practice, we use OLMo 2 1B and the trained OLMo 2 1B SFT to compute the loss difference on target tokens. We use UltraChat and WildChat ( Ding et al., 2023 ; Zhao et al., 2024a ) as our masked perplexity sets.
Task ICL Format Metric Temp Top-p Max toks P@k (n) # sub Base Main Suite GSM8K* ( 2021 ) 8 α CoT EM pass@k 0.6 0.6 512 1, 4 (8) - GSM Symbolic* ( 2024 ) 8 α CoT EM pass@k 0.6 0.6 512 1, 4 (8) 3 Minerva MATH* ( 2022 ) 4 α CoT EM pass@k 0.6 0.6 1024 1, 4 (4) 7 Math MATH 500* ( 2022 ; 2023 ) 4 α CoT EM pass@k 0.6 0.6 1024 1, 16 (32) - HumanEval* ( 2021 ) 3 Code Exec pass@k 0.6 0.6 512 1, 16 (32) - MBPP* ( 2021 ) 3 Code Exec pass@k 0.6 0.6 512 1, 16 (32) - BigCodeBench* ( 2024 ) 3 Code Exec pass@k 0.6 0.6 1280 1 (5) - DS 1000* ( 2022 ) 3 Code Exec pass@k 0.6 0.6 1024 1 (5) - Deepseek LeetCode* ( 2024 ) 0 Code Exec pass@k 0.6 0.6 512 1, 16 (32) - MultiPL-E HumanEval* ( 2022 ) 0 Code Exec pass@k 0.6 0.6 1024 1, 16 (32) 6 Code MultiPL-E MBPP* ( 2022 ) 0 Code Exec pass@k 0.6 0.6 1024 1, 16 (32) 6 HumEval FIM Single* ( 2022 ) 0 FIM pass@1 0.8 0.95 512 1 (10) - HumEval FIM Random* ( 2022 ) 0 FIM pass@1 0.8 0.95 512 1 (5) - FIM HumEval FIM Multi* ( 2022 ) 0 FIM pass@1 0.8 0.95 512 1 (1) - ARC ( 2018 ) 5 MC Acc - - - - 2 MMLU STEM ( 2021b ) 5 MC Acc - - - - 19 MedMCQA* ( 2022 ) 5 MC Acc - - - - - MedQA* ( 2021 ) 5 MC Acc - - - - - STEM QA SciQ* ( 2017 ) 5 MC Acc - - - - - MMLU Humanities ( 2021b ) 5 MC Acc - - - - 13 MMLU Social Sci. ( 2021b ) 5 MC Acc - - - - 12 MMLU Other ( 2021b ) 5 MC Acc - - - - 14 CSQA ( 2019 ) 5 MC Acc - - - - - PiQA ( 2020 ) 5 MC Acc - - - - - SocialIQA ( 2019 ) 5 MC Acc - - - - - DROP Gen2MC* (§ A.4.2 ; 2019 ) 5 MC Acc - - - - - Jeopardy Gen2MC* (§ A.4.2 ; 2024 ) 5 MC Acc - - - - - NaturalQs Gen2MC* (§ A.4.2 ; 2019 ) 5 MC Acc - - - - - SQuAD Gen2MC* (§ A.4.2 ; 2016 ) 5 MC Acc - - - - - CoQA Gen2MC* (§ A.4.2 ; 2019 ) 0 † MC Acc - - - - - Non-STEM QA Basic Skills* (§ A.4.2 ) 5 MC Acc - - - - 6 HellaSwag ( 2019 ) 5 RC per-char {}_{\text{per-char}} Acc - - - - - WinoGrande ( 2020 ) 5 RC none {}_{\text{none}} Acc - - - - - Lambada ( 2016 ) 0 RC per-char {}_{\text{per-char}} Acc - - - - - Basic Skills* (§ A.4.2 ) 5 RC per-token {}_{\text{per-token}} Acc - - - - 6 DROP ( 2019 ) 5 GenQA F1 0 1 100 - - Jeopardy ( 2024 ) 5 GenQA F1 0 1 50 - - NaturalQs ( 2019 ) 5 GenQA F1 0 1 50 - - SQuAD ( 2016 ) 5 GenQA F1 0 1 50 - - GenQA CoQA ( 2019 ) 0 † GenQA F1 0 1 50 - - Base Held-out Suite MMLU Pro ( 2024a ) 5 MC Acc - - - - 13 LBPP* ( 2024 ) 0 Code Exec pass@k 0.6 0.6 4096 1 (32) - Deepmind Math* ( 2019 ) 5 CoT EM pass@k 0.6 0.6 2048 1 (1) - BigBench Hard ( 2022 ) 3 CoT EM Acc 0.6 0.6 512 1 (1) 55
### A.5 Base Model Additional Decontamination Details
Important: this section is adapted from the documentation of the decon package; for up to date information, please consults the official documentation: github.com/allenai/decon/doc/simple-details.md
Evals provide measurable outcomes for model capabilities. We hope that these are meaningful measurements. When evals leak into training data we run the risk of overfitting on evals.
#### A.5.1 Definitions and Preliminaries
Training data and evals both consist of variable length token sequences. Contamination is a sufficient presence of a given eval sequence e e in a given training sequence t t .
We characterize the problem as an approximate substring search for e e in t t for all e ∈ E , t ∈ T e\in E,t\in T .
Our goal is to partition the set T × E T\times E into the set of contaminated documents, denoted as C C , and the set of pure documents, denoted as P P .
We note that | T | ≫ | E | |T|\gg|E| and generally C C is very sparse within T T , as | C | ≪ | P | |C|\ll|P| .
Our goal is to call whether any training sequence t t is derived directly from an eval sequence e e . This involves distinguishing direct derivation of t t to e e from both noise and any source material for e e .
#### A.5.2 Example of Contamination
There is great diversity in the format and purpose of evaluation suites.
decon is fundamentally counting tokens, so it does not consider the intent or semantics of eval instances. But it does leverage the inherent structure of evals to better distinguish between sequences that originate from source material and those that are derived directly from evals.
Knowledge evals frequently have shorter answers.
Reasoning evals frequently have longer answers with a much larger sets of potential token sequences.
Retrieval evals frequently have a substantial passage from source material which acts as an almost input to a program selected by the question component.
#### A.5.3 Eval Normalization
decon normalizes all eval instances into question (Q), answer (A), and passage (P) components. A given eval split may hold out an answer and may or may not contain a passage depending on the task.
An eval instance can be described as having a Q, QA, QP, or QAP composition.
• Question All eval instances to be decontaminated contain a question, and it serves as the primary vessel for information to describe the task. decon uses the question field for initial identification of contamination clusters. Questions with substantial information content and a strong match are sufficient to call contamination.
• Answer While the answer of an eval is important for measuring whether a model has learned a specific task, in the context of decontamination the answer primarily serves to provide supporting evidence of contamination. This is particularly important for questions with low information content or those that have edits.
• Passage The passage, often derived from reference documents, is not a strong indicator of contamination, but in conjunction with a substantial question and answer match, further supports a contamination call.
#### A.5.4 Decon Implementation
We can now describe a computational tractable definition of contamination. We start with the simplest scenario, evals that only have a question component Q, and later extend the approach for QA, QP, and QAP scenarios.
##### Detecting contamination
Scoring segments of training documents against evals is somewhat problematic because there is substantial variation in the length of eval and training documents.
##### Cluster discovery
We start by defining a contamination cluster as a substring of a training document and a set of candidate evals which have at least 1 matching ngram.
We discover clusters by sequentially sampling training document ngrams and checking for a hit in an inverted index which resolves ngrams to eval document ids. Upon an initial hit we expand left and right from the initial hit index until we observe a certain number misses, representing inserts, deletions, or edits.
The initial hit produces a set of matching document ids, which we call the active set. Each subsequent ngram lookup on traversal produces a set of matching documents from the inverted index, which we call the step set. We use the intersection between the active set and the step set to identify which documents in the active set hit for a given step. Once a specific document reaches 11 misses, it is removed from the active set. We repeat this process until the active set is empty or we reach the training document boundaries. At each step we accumulate the unique ngrams matched scoped by eval document id. The end result is a map of document ids to a set of unique ngram shingle matches.
##### IDF-weighted overlap
Contamination scoring uses inverse document frequency (IDF) weighting:
O = ∑ x ∈ U t ∩ U e idf ( x ) ∑ y ∈ U e idf ( y ) O=\frac{\sum_{x\in U_{t}\cap U_{e}}\text{idf}(x)}{\sum_{y\in U_{e}}\text{idf}(y)}