Olmo 3 — part 14/19

Cleaned-text code points [202849, 218665). Bibliography omitted here; source.md includes it.

• For data curation infrastructure and experimentation: Alexander Wettig, Allyson Ettinger, Amanda Bertsch, Bailey Kuehl, David Heineman, Ian Magnusson, Jake Poznanski, Jiacheng Liu, Kyle Lo, Luca Soldaini, Matt Jordan, Mayee Chen, Tyler Murray, Tyler Romero

• For evaluation methodology and infrastructure: Akari Asai, Alexander Wettig, David Heineman, Dustin Schwenk, Hamish Ivison, Harsh Trivedi, Ian Magnusson, Kyle Lo, Luca Soldaini, Maarten Sap, Malia Morgan, Pradeep Dasigi, Regan Huff, Robert Berry, Ronan Le Bras, Rulin Shao, Saumya Malik, Saurabh Shah, Shannon Zejiang Shen, Shashank Gupta, Tyler Murray, Victoria Graf, Yuling Gu

• For mid- and post-training data curation and experimentation: Akari Asai, Alisa Liu, Allyson Ettinger, David Graham, David Heineman, Faeze Brahman, Hamish Ivison, Harsh Trivedi, Jacob Morrison, Kyle Lo, Lester James V. Miranda, Luca Soldaini, Matt Jordan, Michael Noukhovitch, Nathan Lambert, Pradeep Dasigi, Rui Xin, Saurabh Shah, Scott Geng, Saumya Malik, Shashank Gupta, Shuyue Stella Li, Teng Xiao, Valentina Pyatkin, Victoria Graf, Yapei Chang, Zhiyuan Zeng

• For compute infrastructure setup and support: Michael Schmitz, Michael Wilson, Michal Guerquin, Sam Skjonsberg, Tucker Wilde

• For mentorship, advising, program management, and broader strategy: Ali Farhadi, Ashish Sabharwal, Hannaneh Hajishirzi, Luke Zettlemoyer, Noah A. Smith, Pang Wei Koh, Taira Anderson

• For technical leadership and cross-workstream contributions: Hannaneh Hajishirzi, Kyle Lo, Luca Soldaini, Nathan Lambert, Pradeep Dasigi

Authorship for this work was determined by those making direct contributions to the Olmo 3 models, related artifacts, and their release. Core contributors are recognized for their sustained, significant contributions critical to the success of the Olmo 3 project.

## Acknowledgments

This research used resources of the Oak Ridge Leadership Computing Facility, which is a DOE Office of Science User Facility supported under Contract DE-AC05-00OR22725. We acknowledge the National Artificial Intelligence Research Resource (NAIRR) Pilot and Microsoft Azure for contributing to the results in this work. We are grateful for feedback throughout our development process from the open source language model developer community, especially those from Common Pile/Comma, SmolLM3, Marin, Apertus and Gaperon.

### A.1 Base Model Additional Training Details

Table 34 summarizes modeling configuration for Olmo 3 7B and Olmo 3 32B. Table 35 provides overview of training hyperparameters during the three stages of base model development: pretraining, midtraining, and long-context extension. Table 34 describes parallelism configuration for the stages, and lists measured throughput in tokens per second (TPS) for each. Finally, Figure 28 shows training cross entropy loss and gradient norm for Olmo 3 Base 7B and 32B during the pretraining stage.

### A.2 Base Model Additional Data Details: Pretraining

#### A.2.1 CommonCrawl

The majority of our pretraining corpus comes from CommonCrawl ( Common Crawl Foundation, ) . We start with 104 dumps, starting with CC-MAIN-2013-20 and ending with CC-MAIN-2024-51 , roughly covering dates from mid-2013 until late 2024. We linearize the WET files provided by Commoncrawl using Resiliparse, yielding an initial pool composed of 252.6B documents.

Next we apply a pipeline of heuristic filtering steps to further prune down the dataset to a size amenable for pretraining. Our steps essentially follow those of DCLM ( Li et al., 2024a ) , with a few small differences. We start with URL-based filtering, identifying and removing documents that have URLs containing banned words or subwords from the blacklists used by FineWeb ( Penedo et al., 2024 ) and RefinedWeb ( Penedo et al., 2023 ) . This step removes roughly 1% of the data pool. Then we apply the DCLM collection of heuristic filters, roughly targeting and removing: i) very short documents, ii) very long documents, iii) documents with not enough alphanumeric characters, and iv) documents with large amounts of internal repetition. Next, we modify and remove any lines or paragraphs in each document that have i) too many numeric characters or ii) any boilerplate phrases such as "items in cart" or "read more…", and then we fully remove any documents that have been obliterated by these line-specific removals. We then apply a FastText English language filter, mirroring DCLM and using a threshold of 0.65 to identify documents as containing English text. Finally, we apply a subset of the rules for identifying questionable sentences from MADLAD-400 ( Kudugunta et al., 2023 ) . Ablation tests show that only rules 2 and 5 from MADLAD improve dataset quality, targeting sentences that have a large number of capitalized words or contain a "cursed regex". If the number of sentences in the document is less than 5 or if at least 20% of sentences are questionable, we remove the document from the corpus.

Overall, the heuristic steps remove 76% of the total pool, and the English filtering step removes an additional 2.5% of the pool. This leaves a pool of 38.7B documents, attaining a survival rate of 15.1%. While each of these described steps is incorporated into the DCLM processing pipeline, we note that these heuristic filters are commutative and that the English filtering is the slowest step, so efficiency gains can be attained by putting the language-filtering step at the end. We spent a total of 1030 i4i.32xlarge EC2 hours in this step, incurring a cost of approximately $11,300. An exact breakdown of how much time was spent in each step is provided in Table 36 .

#### A.2.2 Deduplication

As described in the main paper, we apply a three-stage deduplication pipeline to our dataset, with each stage targeting progressively more nuanced forms of redundancy: (i) global exact deduplication based on document content hashes to remove identical copies, (ii) 32-way sharded MinHash deduplication with exact Jaccard similarity verification to remove near-duplicate documents, and (iii) 56-way sharded fuzzy suffix array deduplication to eliminate repeated boilerplate text. We note that while applying exact deduplication before MinHash deduplication is technically redundant, exact deduplication is substantially more efficient computationally; hence this two-pass approach is much faster overall. For the exact and MinHash deduplication stages, we utilize the Duplodocus tool, 59 59 59 github.com/allenai/duplodocus and for the suffix array deduplication stage, we employ bsade . 60 60 60 github.com/liujch1998/bsade

##### Exact Deduplication

We perform exact deduplication in two sequential passes. During the heuristic filtration pipeline, we annotate each document with a 128-bit hash computed from the document text. We then apply an initial deduplication step to each of the 104 processed CommonCrawl dumps individually, arbitrarily retaining one copy of each document per dump. This within-dump deduplication removes 24% of the surviving document pool.

Following this, we aggregate all documents globally and perform a second exact deduplication pass across the entire corpus, again arbitrarily keeping one copy of each document. This global pass removes an additional 43% of the surviving pool. In total, exact deduplication eliminates 66% of the input documents, reducing the corpus to 12.7 billion documents for subsequent MinHash processing.

##### MinHash Fuzzy deduplication

We partition the 12.7 billion document corpus resulting from exact deduplication into 32 shards of approximately equal size and perform MinHash deduplication independently on each shard. Our MinHash procedure broadly follows the approach outlined in ( Lee et al., 2022 ) . We tokenize documents using the p50k tokenizer and construct sets of 5-gram token sequences. We then apply a MinHash locality-sensitive hashing scheme with 26 bands of size 11, configured to target a Jaccard similarity threshold of 0.80.

For any pair of documents that share at least one matching bucket, we treat them as connected by an edge in graph-theoretic terms. We construct a graph from the union of all such edges and identify connected components within this graph. Each document in a connected component is then annotated with a unique identifier for that component.

In a second verification phase, we explicitly compute pairwise Jaccard similarities within each MinHash-identified cluster to eliminate false positives. For this verification, we use 3-gram token sequences. Our approach varies based on cluster size: for connected components containing 500 or more documents, we apply a more stringent MinHash configuration using 200 bands of size 31; for components with fewer than 500 documents, we perform exhaustive pairwise Jaccard similarity checks and generate final duplicate clusters from these results.

After annotating all documents according to their true Jaccard similarity with other documents in the corpus, we retain only the most recent version of each document based on crawl date, removing all earlier duplicates. This complete MinHash deduplication procedure eliminates 24% of the input documents, leaving 9.8 billion documents in the pool.

##### Suffix Array deduplication

In the final deduplication stage, we employ suffix arrays to identify and remove substrings that appear repeatedly throughout the dataset. We partition the 9.8 billion document corpus into 56 shards of roughly equal size and run suffix array deduplication independently on each shard.

For each shard, we construct a suffix array and identify every byte sequence of length 500 or greater that appears at least twice in the shard. We then apply a novel “fuzzy suffix array” removal strategy that considers contiguous text spans within each document. Specifically, if a text span is bounded on both sides by 500-byte sequences that appear multiple times in the suffix array, and at least 80% of the span is covered by such repeated sequences, we remove the entire span. This strategy targets cases where naive suffix array deduplication would leave short, unique fragments interspersed between removed substrings. For text that does not meet this bookended criterion, we remove all individual occurrences of repeated 500-byte sequences.

After these three rounds of deduplication—exact, MinHash, and suffix array—we arrive at a final corpus of 9.7 billion documents.

#### A.2.3 Topic Classification

After strict rounds of deduplication, we partition our data according to topic using the 24 topic categories introduced in WebOrganizer ( Wettig et al., 2025 ) . Rather than using the 140M parameter topic classifier used by WebOrganizer, we train a FastText classifier 61 61 61 huggingface.co/allenai/dolma3-fasttext-weborganizer-topic-classifier to support cost-effective topic classification at scale. To train this classifier, we use the Llama-labeled training data used to train the original WebOrganizer category as well as an extra 506,746 examples with topics labeled by a combination of gpt-4.1 and o4-mini. The performance of this classifier is outlined in Table 37 .

#### A.2.4 CommonCrawl Mixing

We perform a hierarchical mixing procedure on our data. Our procedure Olmix ( Chen et al., 2026 ) generates prescriptions for which percentage of the training mix should come from each topic or source, but offers no guidance on the quality composition within each topic. While prior works, such as DCLM ( Li et al., 2024a ) use a quality classifier to flatly filter data as high-quality or not, we take a more fine-grained approach and perform selective up and down-sampling within each WebOrganizer topic depending on the quality signal. This section formalizes the search procedure we use to generate these upsampling curves.

##### Problem formulation

We discuss this procedure in more general terms: consider a category with X X tokens, partitioned into Q Q strictly ordered quality buckets, where the q t ​ h q^{th} bucket contains X q X_{q} tokens. Further assume that Olmix prescribes that Z Z tokens be taken from this category, and that at no point do we want to upsample any quality bucket more than M M times. This equates to a search problem, where we need to take Z q Z_{q} tokens from the q t ​ h q^{th} bucket such that ∑ q Z q = Z \sum_{q}Z_{q}=Z and ∀ q , Z q / X q ≤ M \forall q,\;Z_{q}/X_{q}\leq M .

##### Parameterizing the solution space

To reduce the dimensionality of this search space, we make a modeling choice, where we search over a family of functions that control the upsampling ratio that meets the following criteria: • Every function in the family is convex and monotonic.

• The functions are defined on the interval [ 0 , 1 ] [0,1] , which can be normalized to the token counts later.

• We are able to control the integral such that ∫ 0 1 f ⁡ ( x ) ​ 𝑑 x = Z / X \int_{0}^{1}f(x)dx=Z/X .

• We can control the maximum average value of any one bucket. Suppose the q t ​ h q^{th} bucket of data is arranged on the x x -axis from [ a , b ] [a,b] , then the maximum upsampling constraint correlates to the inequality 1 b − a ​ ∫ a b f ⁡ ( x ) ​ 𝑑 x ≤ M \frac{1}{b-a}\int_{a}^{b}f(x)dx\leq M .

• We have the option to filter out the lowest quality buckets, i.e. ∫ 0 a f ⁡ ( x ) ​ 𝑑 x = 0 . \int_{0}^{a}f(x)dx=0.

One such family of functions that meets these criteria is the family of truncated power-exponential functions: f p , λ ​ ( x ) = { 0 , for ​ x < a C ​ ( x − a ) p ⋅ e λ ⁡ ( x − a ) , for ​ x ≥ a f_{p,\lambda}(x)=\begin{cases}0,&\text{for }x<a\\ C(x-a)^{p}\cdot e^{\lambda(x-a)},&\text{for }x\geq a\\ \end{cases} Specifically, this becomes a feasibility problem for each topic of the data, where we search over parameters p , λ , C p,\lambda,C such that the constraints • (Token yield is satisfied) ∫ 0 1 f p , λ ​ ( x ) ​ 𝑑 x = Z / X \int_{0}^{1}f_{p,\lambda}(x)dx=Z/X .

• (Maximum upsampling ratio is honored) 1 b ​ ∫ 1 − b 1 f p , λ ​ ( x ) ​ 𝑑 x ≤ M . \frac{1}{b}\int_{1-b}^{1}f_{p,\lambda}(x)dx\leq M.

• (Function is monotonic) λ ≥ 0 . \lambda\geq 0.

are satisfied. The maximum upsampling constraint has been simplified such that, assuming monotonicity, the most upsampled quality bucket would be the highest-quality one, with an assumed data proportion of b b .

##### Implementation details

For each WebOrganizer topic, we set the maximum upsampling ration to be M = 7 M=7 and also throw away the bottom 40 % 40\% in terms of quality, a = 0.40 a=0.40 . Then we numerically solve for feasible p , λ , C p,\lambda,C . If the q t ​ h q^{th} quality bucket spans from the q − q^{-} percentile to the q + q^{+} percentile of the data, then the upsampling ratio for this bucket of data should be 1 q + − q − ​ ∫ q − q + f ⁡ ( x ) ​ 𝑑 x \frac{1}{q^{+}-q^{-}}\int_{q^{-}}^{q^{+}}f(x)dx .

#### A.2.5 Validating Quality Upsampling and Mixing

We validate our quality upsampling curves and mixing methodology both individually and jointly using small-scale 1B parameter models trained on 100B tokens. Our validation consists of three experiments:

##### Targeted mixing

We first verify that our mixing methodology can successfully optimize for specific prediction targets. Using our swarm optimization procedure, we create mixes optimized for three different objectives: the QA average, Math average, and Code average from OlmoBaseEval . We compare these targeted mixes against both the natural data distribution and the final Olmo 3 mix. Table 39 demonstrates that our swarm optimization successfully adapts the data distribution to match specific capability targets. While the final OlmoBaseEval mix exhibits slightly higher (worse) BPB scores than task-specific mixes due to necessary trade-offs across multiple objectives, it substantially outperforms the natural distribution.

##### Quality-aware upsampling