Olmo 3 — part 4/19

Cleaned-text code points [45474, 61394). Bibliography omitted here; source.md includes it.

We start with 104 dumps from the CommonCrawl corpus, with a cutoff date of December 31, 2024. Following DCLM ( Li et al., 2024a ) , we remove HTML artifacts and extract the semantic text from WARC files using Resiliparse ( Bevendorff et al., 2018 ) . Where applicable, we directly leverage the raw Resiliparse-extracted data from DCLM-pool 6 6 6 data.commoncrawl.org/contrib/datacomp/DCLM-pool/index.html ( Li et al., 2024a ) and apply Resiliparse extraction on dumps not contained with the DCLM-pool.

##### Heuristic filtering

We apply a pipeline of heuristic filtering steps to prune our initial collection of 252.6B documents to a size amenable for pretraining. Our process closely follows that of DCLM ( Li et al., 2024a ) with minor modifications to improve data quality and computational efficiency. We first apply URL filtering to remove spam and adult-content from an expanded blocklist. We then remove documents that were either too short or too long, followed by filtering documents that contain excessive symbols or insufficient quantities of alphabetic characters. Next we remove documents containing large amounts of internal repetition and apply filtering to remove common spam phrases, fully removing any documents that are identify by these heuristics. We then use a fastText classifier 7 7 7 lid.176 from fasttext.cc/docs/en/language-identification to identify the language of each document, keeping only documents that contain English text. As a final step, we apply sentence-level heuristics from Madlad400 ( Li et al., 2024a ) . In aggregate, this process reduces the size of our data pool by 84.6%, yielding a corpus of 38.8B documents. More details are provided in Appendix § A.2 .

##### Deduplication

The web data we collect from CommonCrawl naturally contains an abundance of duplicated documents. This duplication arises from repeated crawls of the same website, near-copies of documents appearing across multiple web pages, and highly-repeated boilerplate text. Our deduplication strategy is motivated by three observations from prior work: 1) deduplication generally leads to more token-efficient training ( Lee et al., 2022 ) ; 2) duplicate count serves as a weak signal of data quality, with higher duplicate counts indicating higher quality ( Fang et al., 2025a ) ; 3) repeating documents more than a handful of times provides rapidly diminishing returns ( Muennighoff et al., 2025a ) .

Given these observations, we design our deduplication strategy to enable a future quality-based upsampling step (Section 3.4.4 ). We aggressively deduplicate our dataset at multiple granularities, targeting the removal of exact replicas, near-duplicates, and repeated filler text. While this necessarily discards the quality signal from duplicate counts, it produces a clean base dataset from which we can later selectively reintroduce repetition for high-quality documents. Our goal is a final dataset with minimal repetition overall, with any duplication concentrated in high-quality data. We implement our deduplication procedure in three distinct stages:

1. Exact deduplication We apply global deduplication based on document text hashes to remove all exact copies. This step identifies 67% of the pool as duplicates, reducing the dataset from 38.7B to 12.8B documents.

2. Fuzzy deduplication We apply MinHash-based deduplication to identify and remove near-identical documents, such as documents copied across multiple domains that differ only in headers or footers. We partition the dataset into 32 shards, ran MinHash deduplication on each shard, then performed exhaustive pairwise Jaccard similarity checks within each identified cluster. From each cluster, we retain the most recent document by crawl date. This procedure identified 23% of the pool as duplicates, yielding 9.8B documents.

3. Substring deduplication The previous steps removes whole duplicate documents but did not address repeated content within individual documents. Many documents contain substantial boilerplate text or HTML artifacts (e.g., headers and footers) of limited training value. To remove these repeated substrings, we apply a novel fuzzy suffix-array-based deduplication procedure. We partition the dataset into 57 shards and apply this procedure to each, marking any substring of 500 or more bytes that occurred multiple times. Unlike previous suffix-array methods, we preserve at least one occurrence of each repeated substring in the corpus. We then merge the intervals marking repeated substrings to also remove short substrings sandwiched between longer repeated segments. This procedure removes 14% of text bytes, yielding 9.7B documents totaling 36.5T bytes of uncompressed text.

This three-stage procedure reduces the web corpus from 38.7B to 9.7B documents—a 75% reduction in document count. The resulting aggressively deduplicated dataset can then be partitioned by topic and quality and controllably upsampled for training.

To scale our deduplication strategy, we develop the Duplodocus tool, 8 8 8 github.com/allenai/duplodocus a native-rust toolkit for large-scale distributed execution of both hash-based exact deduplication and MinHash fuzzy deduplication.

##### Topic and quality classification

We use our WebOrganizer tool ( Wettig et al., 2025 ) to partition the deduplicated corpus into 24 topics (e.g., “ Adult Content ”, “ Politics ”, or “ Science and Technology ”). To speed up processing of the Dolma 3 pool, we distill the transformer-based models by Wettig et al. 2025 into a simpler fastText model. 9 9 9 huggingface.co/allenai/dolma3-fasttext-weborganizer-topic-classifier We only partition by topic, not format. We also train and apply a fastText-based quality classifier 10 10 10 huggingface.co/allenai/dolma3-fasttext-quality-classifier to assign each document a quality score. Following DCLM ( Li et al., 2024a ) , we use OpenHermes-2.5 ( Teknium, 2023 ) and ELI5 ( Fan et al., 2019 ) as positive training examples, supplemented with UltraChat-200k ( Ding et al., 2023 ) and WildChat-1M ( Zhao et al., 2024a ) . Negative training examples consist of 30GB sampled from DCLM-RefinedWeb.

We apply both the topic and quality classifiers to the full deduplicated corpus in order to partition the dataset. Documents are first partitioned by topic, then within each topic partition we compute quality score percentiles and subdivide documents into vigintile buckets (5-percentile intervals). This two-stage partitioning yields 480 disjoint subsets (24 topics × \times 20 quality tiers), enabling fine-grained control over the topic and quality distribution of our pretraining mixture.

##### Final web data pool

The above steps results in an 8T-token pool of annotated data, partitioned into buckets according to topic and text quality. This pool serves as the foundation for our pretraining mixture, though additional processing is required to construct the final training data. Specifically, we apply quality-based filtering and topic reweighting to generate a balanced, high-quality mixture, as discussed in Section § 3.4.4 .

#### 3.4.2 Preparing our olmOCR science PDFs Data Pool

We curate a novel dataset of academic PDFs, replacing our previous use of peS2o ( Soldaini and Lo, 2023 ) . These documents are crawled “politely”: we identify our crawler as AI2Bot , 11 11 11 Crawling notice: allenai.org/crawler we adhere to robots.txt , and do not bypass paywalls. The crawler is seeded with a focus on academic sites and paper repositories. We process all PDFs using the first version of olmOCR ( Poznanski et al., 2025a ) . Ultimately this crawl generates a collection of 238 million unique PDF documents with a cutoff date of December 2024.

##### olmOCR text extraction

To convert PDFs to a format usable by our trainer, we apply pre-filtering and text extraction. If a document contains born-digital text, we used the Lingua language detector to retain only English documents and remove documents where spam or SEO-optimization keywords exceeded 0.4% of total words. We then extract text using olmOCR ( Poznanski et al., 2025a ) (versions 0.1.49-0.1.53). If olmOCR fails, we use Poppler’s pdftotext as a fallback; documents requiring this fallback for more than 1 in 250 pages are excluded from the corpus. This yields a dataset of 160 million PDF documents.

##### Deduplication

We then identify and remove any fuzzy-duplicates using a MinHash algorithm. This differs slightly from the MinHash step we apply to the web text corpus in Section § 3.4.1 : we use the MinHash parameters as in FineWeb ( Penedo et al., 2024 ) , which targets document pairs with at least 75% similarity; and we omit an exhaustive pairwise Jaccard similarity check. After this deduplication step, we were left with a corpus of 156M documents for a removal rate of 2.3%.

##### PII filtering

Next we remove documents containing PII from the pool of PDFs. Our goal was to to remove documents that contained sensitive standalone PII, such as government IDs and login information, as well as documents that link biographical, medical, location, employment, or educational information to a specific individual. Through iteration, we determine that PII detection must be document type-aware to be effective. For example, a conference paper might contain name and place of employment of authors; however, as research articles are intended for publication, removal would not make sense. At the same time, a bank statement might contain the same name and employer information, and is clearly a document a language model should not be trained on. The rule we follow is: is this document type intended for public dissemination? We use manual annotators to iterate which documents types are not suitable for public dissemination, and what PII attributes we should consider. The resulting taxonomy is used as part of a multi-stage model-based PII filtering pipeline.

First we classify documents using a prompt to Gemma 3 12B ( Gemma 3 Team, 2025 ) on the first page of each document to determine if they contain any sensitive standalone PII, or link sensitive information to an individual. Next, we use Gemma 3 4B on the first 5,000 characters of each document to arrive at a set of flags describing the type of document. From these classification results, we develop a set of rules to identify which types of documents containing PII should be publicly available and which should be filtered. Ultimately this removes 4.9% of the remaining pool and yields a pool of 148 million documents. See Poznanski et al. (2025a) for more a complete overview of the PII removal pipeline.

##### Heuristic filtering

After PII removal, we apply a round of heuristic filtering to further remove low-quality documents. Filters applied in this step include checking for: non-English documents not originally caught by the Lingua filter; documents that were more than 30% tables; and documents that contain more than 20% numbers. Next we apply modifications that convert markdown tables to HTML and remove URL references. The combination of these filtration steps yield a corpus of 108 million documents. This corpus is then partitioned into 24 topical buckets, according to the WebOrganizer topic classifier ( Wettig et al., 2025 ) , and passed off to the mixing (Section § 3.4.4 ).

#### 3.4.3 Preparing Code, Math, and other sources

##### Code

For code data, we use Stack-Edu ( Allal et al., 2025 ) , an improved curation of GitHub repositories from the-stack-v2 dataset ( Lozhkov et al., 2024 ) with additional filtering for educational programming content. We keep partitions of the data by programming language for subsequent mixing.

##### Math

As in OLMo 2 , we include arXiv documents from the Proof-Pile-2 dataset ( Azerbayev et al., 2023 ) , which in turn are from the RedPajama dataset ( Together AI, 2023 ) and have a cutoff date of April 2023. We use this source primarily because it preserves the original LaTeX notation, enabling the model to learn both mathematical content and how to properly format it.

Furthermore, we replace our previous use of OpenWebMath ( Paster et al., 2023 ) with FineMath ( Allal et al., 2025 ) , a subset of Common Crawl documents that contain mathematical educational content and have been reprocessed to preserve proper mathematical notation. We include all documents that have a quality score of at least 3 (out of 4), according to the FineMath classifier. This data has a cutoff date of September 2024.

##### Other

Finally, we include the Wikipedia and Wikibooks sources from Dolma ( Soldaini et al., 2024 ) as base sources of encyclopedic knowledge. These are both the “English” and “Simple” editions of Wikipedia and Wikibooks with a cutoff date of March 2023. These sources were processed using WikiExtractor ( Attardi, 2015 ) to remove markup formatting, and all documents with 25 or fewer words were filtered out to exclude template pages or pages that encountered XML parsing errors.

#### 3.4.4 Sampling and Mixing over Data Pools

The data sources described above collectively provide over 9 trillion tokens of diverse text data. Transforming this collection into a training dataset requires a mixing and sampling pipeline to prescribe exactly how much of each source to include in a final training mix, and how much, if any, upsampling to apply to each source. We apply a mixing strategy that draws on swarm-based methods to train and evaluate many smaller proxy models, using these results to inform an optimal mix. Further, we apply a novel conditional mixing procedure to account for the fact that our data sources were being constantly refined and updated throughout the development cycle. In this section, we describe how we derive the final at the mixing ratios for each source; for web text, we only optimize ratios at the topic category level and apply quality-aware upsampling to obtain the final mix.

##### Constrained data mixing

We applied data mixing across all pretraining sources, as well as across the WebOrganizer topics within the web data and PDF sources, and the Stack-Edu programming languages. Our mixing procedure ( Chen et al., 2026 ) , consists of two components: a base procedure that constructs a high-quality mix over a fixed set of data domains, and a meta-procedure called conditional mixing that efficiently updates an existing mix when domains change. Together, these allow us to iteratively build an optimal mix and adapt to data refinements or additions without starting from scratch.

The base procedure follows a swarm-based approach inspired by RegMix ( Liu et al., 2024a ) , Data Mixing Laws ( Ye et al., 2025 ) , and CLIMB ( Diao et al., 2025 ) ; it consists of three stages: 1. Swarm construction . We sample the space of possible mixes by training many small proxy models, each with a different mixing ratio. Specifically, we train 30M-parameter models following the Olmo 3 architecture for 3B tokens (5x Chinchilla), sampling each mix from a Dirichlet distribution centered on the natural (no-mixing) distribution. As a rule of thumb, we launch a swarm of size 5x that of the number of domains. We then evaluate each proxy model on the Base Easy suite.

2. Per-task regression . Each proxy model provides a datapoint mapping mixture weights to task performance—measured in bits-per-byte (BPB)—for each task. We fit a separate generalized linear model for each task, enabling us to predict how any candidate mix will perform.

3. Mix optimization . We find the mixture that minimizes the average task BPB, as predicted by the per-task regression models. Since we ultimately seek a corpus with a 6T token budget, and we avoid repeating any domain more than approximately 4 − 7 4-7 times, this naturally imposes maximum ratio constraints on certain domains based on their available token counts. We solve this constrained optimization using a guided search initialized from a prior or natural distribution.