Puro-2B: Poor Lab's Qwen2-1.5B Trained on RTX 5090 within $5090 — part 5/7

Cleaned-text code points [62401, 78036). Bibliography omitted here; source.md includes it.

We use the same evaluation protocol for the UD and CMA models within each setup. For GSM8K-based SFT and Math&Code SFT with replay, we evaluate the resulting models on GSM8K with the lm-evaluation-harness framework. Because generated answers can differ in format, we use flexible extraction to identify the final answer. We report the average result over three repeated seeds. For Tulu-3 mixed-domain SFT, we use OpenCompass to evaluate the resulting models on 15 tasks for base models and IFEval. The evaluation protocol of the 15 tasks is adapted to the fine-tuned models. This suite covers knowledge, reasoning, code, and instruction following. We report both the overall average and the individual task results, since the effect may differ across tasks. Further evaluation details are provided in Appendix E .

### 3.6 Data Recipe

Domain Phase 1 Phase 2 Tokens Share Tokens Share English 321.31B 73.2% 557.52B 59.4% Math 31.56B 7.2% 171.29B 18.3% Chinese 51.34B 11.7% 88.48B 9.4% Code 34.62B 7.9% 108.11B 11.5% SFT / instruction – 0.0% 12.66B 1.3% Total 438.84B 100.0% 938.06B 100.0%

Our goal is to select a compact set of high-quality and diverse data sources while preserving broad domain coverage. Candidate datasets differ in both quality and capability profile, so source-level metadata alone is not sufficient for data selection. We therefore use proxy benchmarking to compare candidate data slices under a shared protocol. As illustrated in Figure 9 , each run starts from the same Qwen3-0.6B checkpoint and follows the same continuation-training schedule. After continuation training, we evaluate the resulting checkpoint on a fixed 15-benchmark suite. The score vector from this evaluation serves as the capability profile of the candidate slice. These score vectors can guide our data recipe in heuristic. The following three paragraphs describe how we construct fine-grained slices, select and allocate data sources, and balance different capabilities in the final mixture. Detailed settings are provided in Appendix I .

#### 3.6.1 How Do We Select a Good Data Recipe?

##### Fine-grained data slice selection.

We construct candidate data slices according to source size and the availability of sample-level quality scores. We apply three sampling rules. First, if a dataset contains more than 50B tokens and provides usable score labels, we sort its examples by score and sample four local slices around the 0th, 25th, 50th, and 75th percentile positions. Second, for datasets between 5B and 50B tokens, as well as larger datasets without usable score labels, we randomly sample one 4B-token slice. Third, we do not evaluate the remaining smaller datasets. For each large scored source, the four slices form a within-source quantile benchmark. Because these slices come from different positions in the same score ordering, they reveal quality differences that a single source-level result would hide. For example, the q00 slice of DCLM-Dedup ranks third in average proxy performance, while its q25 slice ranks eleventh. This result shows that different regions of the same source can provide substantially different training value.

##### Feature-guided data selection.

We use the measured capability profiles to guide both source selection and data allocation. MegaMath-Web-Pro performs strongly on Math and General, while MegaMath-Code provides the strongest Code signal among the measured candidates. We therefore assign both sources larger shares in Phase 2 to strengthen general and target capabilities. The quantile results also guide within-source filtering. Because the top-scoring region of DCLM-Dedup performs much better than its lower-ranked regions, we retain 34B tokens from this region in Phase 2. This selection corresponds to approximately the top 5% of DCLM-Dedup. FineWeb-Edu-CN ranks highest on Chinese capability, so we also increase its share to preserve Chinese performance. The proxy results provide useful evidence for increasing, reducing, or filtering different data sources. Here, we take heuristics from these capability profiles and still choose the concrete mixing ratios manually.

##### Capability balance across domains.

The proxy results show that different datasets have substantially different capability profiles. A source that performs well on one group of benchmarks may perform less well on others. For example, FineWeb-Edu-CN ranks first on Chinese capability but is weaker on Math, Code, and General ( Appendix I ). We therefore need to balance broad general knowledge with more specialized reasoning capabilities rather than maximize a single benchmark dimension. To make these trade-offs more explicit, we apply PCA to the capability vectors. As shown in Figure 20 , General aligns mainly with PC1, while Code points in the opposite direction. Math aligns mainly with PC2 and remains slightly positive on PC1. In this projection, Code shows a stronger trade-off with General than Math does ( Appendix I ). This observation informs the roles of the two pretraining phases. Phase 1 preserves broad coverage, with English accounting for 73.2% of 438.8B ( Figures 8 and 8 ). Phase 2 then adds more specialist data, but allocates a slightly larger share to mathematics than to code. In this way, the two-phase recipe strengthens the target reasoning capabilities while retaining broad general coverage.

#### 3.6.2 How Do We Preprocess Data and Reproduce the Shards?

##### Data preprocessing pipeline.

The pipeline turns the candidate collection into fixed training shards in four stages. Within-source deduplication removes repeated documents from web components such as DCLM-Dedup and FineWeb-Edu-EN. We do not deduplicate across sources because our analysis found little additional reduction in data volume. Proxy benchmarking evaluates each eligible candidate source or within-source slice under the shared protocol described above and produces a multidimensional capability profile. Heuristic filtering combines these profiles with domain coverage, language balance, and token-budget constraints to determine the retained portion and mixture weight of each source. Final materialization freezes the source revisions, retention rules, mixture weights, and random seeds. It then merges the retained components into fixed shards that are read sequentially during training, without online mixing or sampling. In addition, we also use PreSelect fastText [ SHUM et al., 2025 ] to evaluate the Nemotron-HQ [ Mahabadi et al., 2025 ] and Nemotron-HQ-Synthetic datasets. Apart from this step, we do not perform additional compute-intensive data scoring and primarily rely on existing quality scores.

##### Kaiyuan-Spark preprocessing framework.

We implement the recipe with Kaiyuan-Spark , a Spark-based data preprocessing framework [ Luo et al., 2025a ] . The framework is designed for efficient large-scale data processing. For compute-intensive operations such as MinHash deduplication, it uses native C++ kernels through the Chukonu integration. This implementation is inherited from Chukonu, whose original Spark benchmark reports an approximately 2.5 × 2.5\times speedup over the corresponding JVM implementation [ Yu et al., 2021 ] . The framework also supports the curriculum bucket materialization required by the Phase 2 design. Given a fixed component pool and mixture, it assigns normalized progress within each component, aligns the resulting slices into token-sized buckets, and materializes their order using recorded random seeds. The same component pool can therefore produce either an ordered curriculum stream or a uniformly ordered stream by changing only the materialization order. These two data streams support the UD, CD, and CMA recipes in Phase 2 ( Section 3.4 ).

##### Two-phase data and transition schedule.

All training data are materialized as static shards and manifests before training. Production runs therefore do not rely on online scoring or dynamic mixture sampling. The Phase 1-to-Phase 2 transition spans approximately 43.9 43.9 B training tokens. During this transition, the sampling weight of the Phase 1 replay mixture decreases linearly , while that of the Phase 2 mixture increases linearly . The transition therefore consumes approximately 21.9 21.9 B tokens from the Phase 1 replay data and 21.9 21.9 B tokens from the beginning of the Phase 2 data pool. The full Phase 2 component pool contains approximately 938.1 938.1 B tokens, including the 21.9 21.9 B tokens consumed during the transition. After the transition, the remaining 916.1 916.1 B Phase 2 tokens are consumed without Phase 1 replay. Counting the transition, the Phase 2 training stage consumes approximately 960.0 960.0 B tokens in total: 21.9 21.9 B tokens of Phase 1 replay and 938.1 938.1 B tokens from the Phase 2 pool. Together with Phase 1 training, the complete pretraining run consumes approximately 1.4 1.4 T tokens.

## 4 Evaluation

### 4.1 Evaluation Setup

#### 4.1.1 Model Selection

We compare Puro-2B with recent language models of comparable size, primarily in the 1B–3B parameter range. We group these baselines according to the scope of their public releases. Open-weight models make their weights publicly available but do not release the complete training data or recipe, whereas open-recipe models additionally release the data and training artifacts needed to study or reproduce training.

Open-weight models. • Qwen series [ Yang et al., 2024a , Yang et al., 2024b , Yang et al., 2025a , Qwen Team, 2026 ] : We include Qwen2-1.5B, Qwen2.5-1.5B, Qwen3-1.7B-Base, and Qwen3.5-2B-Base, which provide successive generations of strong compact foundation models in the 1.5B–2B range.

• Gemma series [ Rivière et al., 2024 , Gemma Team, 2025 , Gemma Team, 2026 ] : We select Gemma-2-2B, Gemma-3-1B-PT, and Gemma-4-E2B-Base to cover three generations of Google’s lightweight open models.

• Llama 3.2 [ Meta AI, 2024b ] : We include Llama-3.2-3B, the latest pretrained Llama checkpoint available in the compact 1B–3B range.

• LFM2.5 [ Liquid AI, 2025 ] : We include LFM2.5-1.2B-Base as a recent compact base model designed for efficient on-device deployment.

• Falcon-H1 [ Zuo et al., 2025 ] : We evaluate the standard and deeper 1.5B base variants as efficiency-oriented hybrid-model baselines.

Open-recipe models. • Instella [ Liu et al., 2025 ] : Instella-3B was trained entirely on openly available data, with model weights, training data, configurations, and code released to support reproducibility. Despite using fewer pretraining tokens than many contemporary models, it is reported to remain competitive with leading open-weight models at a similar scale.

• OLMoE [ Muennighoff et al., 2025 ] : OLMoE-1B-7B-0125 is a sparse model with 1B active and 7B total parameters. Its release includes intermediate checkpoints, training data, code, logs, and evaluation resources. The model is positioned as a strong performance–efficiency baseline among fully-open mixture-of-experts models.

• Yulan-Mini-2.4B [ Yiwen et al., 2025 ] : Yulan-Mini-2.4B is a 2.4B model trained on 1.1T tokens, with its training recipe and phase-wise data composition publicly documented. It emphasizes data-efficient pretraining and reports competitive performance against similarly sized models trained with substantially more data.

• SmolLM3 [ Bakouch et al., 2025 ] : SmolLM3-3B-Base was pretrained on approximately 11T tokens, with the public data mixture and training configurations released alongside the model. It is presented as a strong fully-open model at the 3B scale, with performance competitive with several larger compact models.

• MiniCPM5 [ OpenBMB Team, 2026 ] : MiniCPM5-1B-Base is a compact dense model developed for local and resource-constrained deployment. Its associated web and mathematical training corpora are publicly released, while the MiniCPM5 family targets strong reasoning, coding, and mathematical capabilities within a small deployment footprint.

For a fair comparison, we consistently evaluate the pretrained/base versions of all models.

#### 4.1.2 Benchmarks

Our evaluation covers 15 benchmarks, organized into two groups: mathematics and code, and reasoning and knowledge. For mathematical reasoning, we use GSM8K [ Cobbe et al., 2021 ] and the original MATH benchmark [ Hendrycks et al., 2021b ] , which measure grade-school arithmetic and competition-level mathematics, respectively. For code generation, we use sanitized-MBPP [ Austin et al., 2021 ] and HumanEval [ Chen et al., 2021 ] , where generated Python programs are verified by unit tests. For reasoning and knowledge, we use MMLU [ Hendrycks et al., 2021a ] , MMLU-Pro [ Wang et al., 2024b ] , ARC-Challenge and ARC-Easy [ Clark et al., 2018 ] , BoolQ [ Clark et al., 2019 ] , CommonsenseQA (CSQA) [ Talmor et al., 2019 ] , HellaSwag [ Zellers et al., 2019 ] , PIQA [ Bisk et al., 2020 ] , SocialIQA (SIQA) [ Sap et al., 2019 ] , WinoGrande [ Sakaguchi et al., 2020 ] , and Big-Bench Hard (BBH) [ Suzgun et al., 2023 ] . Together, these benchmarks assess mathematical reasoning, code generation, academic knowledge, reading comprehension, commonsense understanding, and general multi-step reasoning.

#### 4.1.3 Implementation Details

We evaluate every model with OpenCompass [ Contributors, 2023 ] using a fixed checkpoint and tokenizer revision, dataset snapshot, benchmark-specific prompt, shot examples, generation configuration, and answer postprocessor. GSM8K, MATH, sanitized-MBPP, HumanEval, MMLU-Pro, and BBH use generation-based (GEN) evaluation: the model produces a free-form response from which the final answer or executable code is parsed. GEN decoding is greedy with fixed maximum output and stopping rules. The remaining nine benchmarks use perplexity-based (PPL) evaluation, which ranks the provided candidate answers by their fixed token log-likelihoods and therefore introduces no generation randomness. For GSM8K, a shared deterministic postprocessor prioritizes explicit final-answer markers and boxed answers, preserves signs and fractions, and uses the final numeric expression only when no explicit answer is present.

Following the OLMES [ Gu et al., 2025 ] convention, for tasks supporting both cloze formulation (CF) and multiple-choice formulation (MCF), we evaluate both formulations for each model and use the better-performing formulation when computing the reported aggregate. This selection is performed independently for each model. All scores are percentages. In each table, Avg is the unweighted arithmetic mean over the benchmarks displayed in that table. Throughout the cost–performance and scaling analyses, overall performance P P denotes the unweighted arithmetic mean over all 15 benchmark scores reported in Tables 3 and 4 . Every score in our tables is produced by this common, deterministic pipeline.

### 4.2 Resulting Performance

##### Mathematics and code.

As shown in Table 3 , Puro-2B obtains an average score of 43.50 across the four generative benchmarks. It exceeds Qwen2-1.5B by 3.21 points and is within 4.02 points of Qwen2.5-1.5B. Among the open-recipe baselines, Puro-2B outperforms Instella-3B, OLMoE-A1B/7B, and MiniCPM5-1B-Base, while Yulan-Mini-2.4B, SmolLM3-3B-Base, and the newly included MobileLLM-R1-950M-base achieve higher averages. MobileLLM-R1 is a strong math/code-oriented compact baseline in this panel, exceeding Puro-2B by 3.86 points on the four-task average. Given these differences in model size and capability emphasis, Puro-2B remains a competitive open-recipe model for mathematics and code generation at a comparatively small scale.

##### Reasoning and knowledge.