Puro-2B: Poor Lab's Qwen2-1.5B Trained on RTX 5090 within $5090 — part 6/7

Cleaned-text code points [78036, 93560). Bibliography omitted here; source.md includes it.

Table 4 shows that Puro-2B achieves a mean score of 63.02 over the eleven reasoning and knowledge benchmarks. It outperforms Qwen2-1.5B by 2.48 points and is within 2.51 points of Qwen2.5-1.5B. Within the open-recipe group, Puro-2B surpasses OLMoE-A1B/7B, Yulan-Mini-2.4B, MiniCPM5-1B-Base, and MobileLLM-R1-950M-base, while trailing the larger Instella-3B by only 0.11 points. SmolLM3-3B-Base achieves a higher average, but it is also a larger model. These aggregate results show that Puro-2B remains competitive with larger open-recipe models and retains a 6.29-point reasoning/knowledge advantage over MobileLLM-R1. Together with the reduced training cost enabled by low-precision training, this performance indicates a favorable balance between broad capability and replication cost.

### 4.3 Cost Estimation

#### 4.3.1 Cost-Saving Factors

We estimate four cost factors separately: hardware price efficiency, FP8 throughput, MuonH quality-equivalent compute, and the joint Phase 2 recipe. Each factor uses its own reference comparison, so the reported values are not measurements of one sequential end-to-end speedup. The product of these components can only be treated as illustrative transfers. Hardware uses a specification-and-price accounting proxy, blockwise FP8 uses quality-adjusted throughput with MuonH fixed, and the complete MuonH recipe uses quality-equivalent compute with blockwise FP8 fixed. Phase 2 first fits a scaling law of compute cost using the UD recipe ( Figure 2(b) ), then measures the equivalent costs of the UD recipe for the CD and canonical CMA endpoints to quantify the corresponding gains. These comparisons do not form a joint factorial ablation due to different references, so their products are accounting estimates rather than directly measured end-to-end speedups 9 9 9 For example, MuonH ablation and FP8 ablation use token-per-parameter of 20 (TPP=20) while the production training is an overtrained setup (TPP=700). We run the alternative compute-optimal setup due to limited compute. .

For the ladder evidence below, C = 6 ​ N ​ D C=6ND denotes theoretical training compute, where N N is the scaling parameter count and D D is the number of training tokens. A compute-equivalent multiplier κ \kappa means that a variant trained with C / κ C/\kappa reaches the fitted loss of its baseline at C C . For precision, we define BF16-equivalent compute retention as ρ C = C BF16 / C FP8 \rho_{C}=C_{\mathrm{BF16}}/C_{\mathrm{FP8}} at matched fitted loss; this converts the vertical precision gap into a horizontal compute adjustment.

##### RTX 5090 exhibits a high cost-performance ratio and utilization.

Under the specifications and prices in , the RTX 5090 proxy provides 2.77 × 2.77\times the peak BF16 compute per unit price of H200 and 2.74 × 2.74\times the peak FP8 compute per unit price of H200. Moreover, with training configuration tuning, as shown in Section 3.1.3 , our recipe can achieve 73% MFU in mixed-precision training, making good use of the RTX 5090’s capability.

##### MuonH provides a quality-equivalent compute shift in the matched ladder.

Under the definition above, the quality-equivalent compute reduction is 1 − 1 / κ 1-1/\kappa . The matched scaling ladder in Figure 10(a) estimates a 1.19 × 1.19\times compute-equivalent multiplier for the complete MuonH recipe relative to Muon, corresponding to 16.1% less theoretical compute at matched validation loss. The scaling estimation is performed over TPP=20 ladder with a shared-shape fit to transfer to production run. At the production run of 2B parameters and 1.4T tokens, using this efficiency estimation, the MuonH optimizer can use 1.41 × 10 22 1.41\times 10^{22} FLOPs to match 1.68 × 10 22 1.68\times 10^{22} FLOPs with the Muon baseline.

##### Blockwise FP8 retains a stable quality fraction while accelerating the 2B proxy.

A precision format may change both execution rate and model quality, so its net efficiency multiplier is the product of throughput speedup and effective-compute retention. As summarized in Figure 10(b) , blockwise FP8 has a small validation-loss gap at all five ladder scales. The shared-shape fit gives ρ C = 98.0 % \rho_{C}=98.0\% . Matching BF16 quality therefore requires approximately 2.0% additional nominal compute. At 1.7B, the closest throughput proxy to the 2B production model, median throughput improves by 1.36 × 1.36\times ; charging the quality penalty yields a 1.34 × 1.34\times net speedup, or 25.2% fewer GPU-hours at matched quality. Then, by this estimation, the 2B-parameter, 1.4T-token canonical run would consequently use 22,654 FP8 GPU-hours to achieve an equivalent performance using approximately 30,286 BF16 GPU-hours.

##### The Puro Cost Scaling Law summarizes recipe-specific scale-down behavior.

Starting from the same Phase 1 checkpoint, we continue Phase 2 training with UD at several budgets and use these runs to characterize the empirical cost–performance relationship, which we refer to as the Puro Cost Scaling Law. It is a scale-down relationship for this recipe, not a universal law across model families. Here, we focus on the scale-down trend to help the community with limited compute and time budgets decide a feasible performance target. We fit P = a + b ​ log 2 ⁡ ( C − C P1 ) P=a+b\log_{2}(C-C_{\mathrm{P1}}) , where P P is the unweighted arithmetic mean over the 15 benchmarks reported in Tables 3 and 4 , C C is total reproduction cost, and a a and b b are the fitted intercept and slope. The checkpoint-level values of P P used in the fit are reported in Table 18 . Costs shown in the figure use the USD conversion in Table 5 . We set C P1 = USD ​ 1.84 C_{\mathrm{P1}}=\text{USD }1.84 K from the measured Phase 1 active-training GPU-hours and hold it fixed while fitting performance against the incremental Phase 2 cost. The observed UD checkpoint at approximately $4.4K already exceeds Qwen2-1.5B without CMA. This shifted fit reduces in-sample RMSE from 0.452 for the unshifted log-cost fit to 0.209. Inverting it at the CD endpoint score places its UD cost equivalent at approximately $11.36K, or 1.65 × \times the production recipe’s $6.89K rental-equivalent estimate from measured active-training GPU-hours. Inverting the same fit at the reported Puro-2B aggregate, which additionally includes a constant-LR continuation and averages the last six checkpoints, gives approximately $16.55K, or 2.40 × \times the production cost. The canonical CMA endpoint optimizes ordering, continuation, and checkpoint averaging together. We therefore report its fitted cost equivalent as a recipe-level point estimate rather than as a curriculum-only gain or a universal cost law.

#### 4.3.2 Cross-Model Reproduction Cost

##### Estimation protocol.

For Figure 1 , we prioritize reported monetary cost or accelerator usage. We use reported GPU-hours, or GPU-hours precisely derived from other reported statistics, and convert them with the reference rental rates listed in Table 5 in the appendix. When no such GPU-hour basis is available, we use the token-based route and estimate compute as C = 6 ​ N ​ D C=6ND , using the reported accelerator and MFU when available and H100-equivalent GPU-hours at 70% MFU otherwise. Models that disclose neither source are omitted. For Puro-2B , we use measured active-training GPU-hours: 6,009.46 GPU-hours for Phase 1 and 16,504.95 GPU-hours for Phase 2. We convert their sum using the RTX 5090 rate in Table 5 . Across all models, this hierarchy distinguishes reproduction-cost estimates based on reported monetary costs or accelerator usage ( reported compute ) from token-based estimates ( estimated compute ). The complete coordinate construction, including the treatment of special cases, and the Pareto-frontier procedure are given in Appendix B .

##### Pareto-frontier result.

Under the stated accelerator-cost assumptions, Puro-2B appears above and to the left of the comparison-model frontier in Figure 1 , combining higher average performance with a lower estimated replication cost. This advantage is clearest relative to the open-recipe models.

### 4.4 Post-Training Results and Analysis

We conduct an ablation study of the two Phase 2 designs to examine how the pretraining recipe affects the results after SFT. The uniform data ordering with learning-rate decay (UD) recipe trains the model on globally reshuffled Phase 2 tokens, while the curriculum model averaging (CMA) recipe follows a data curriculum, keeps the learning rate constant near the end of training, and averages the final checkpoints ( Section 3.4 ). We apply the same SFT procedure to the two pretraining checkpoints and compare the resulting UD-based and CMA-based models. The experimental setup is detailed in Section 3.5 and we analyze the results in this section.

##### GSM8K-based SFT.

The GSM8K-based SFT experiment compares the UD and CMA checkpoints without pretraining replay. After SFT, the UD-based model reaches 66.89% mean GSM8K accuracy, while the CMA-based model reaches 68.66%. On average over three random seeds, the CMA-based model is ahead by 1.77 percentage points ( Figure 11(a) and Table 12 ). We use flexible answer extraction to limit score differences caused by output formatting. We then compare the sets of questions answered correctly by the two models. Under this protocol, the CMA-based model still solves more questions that the UD-based model misses. The remaining advantage is therefore better explained by stronger mathematical capability than by answer formatting.

##### Math&Code SFT with replay.

The Math&Code SFT with replay experiment extends the comparison to a larger mathematics mixture, together with code data and a small replay component. It also uses a longer SFT process. One possible explanation for the base-model difference is a temporary advantage from placing higher-quality data near the end of pretraining. A longer SFT process on a different data distribution tests whether this advantage persists. After this longer SFT process, the UD-based model reaches 74.10% mean GSM8K accuracy, while the CMA-based model reaches 76.12%. On average over three random seeds, the CMA-based model is ahead by 2.02 percentage points ( Tables 3 and 13 ). The CMA advantage therefore remains after more SFT updates on a broader data distribution. The gap is also slightly larger than in the GSM8K-based SFT setting.

##### Tulu-3 mixed-domain SFT.

The first two setups focus primarily on mathematics. To examine whether the difference extends beyond this domain, we use Tulu-3 mixed-domain SFT and evaluate the resulting models on 15 benchmarks ( Tables 3 and 4 ). As shown in Figure 11(b) , the CMA-based model scores 1.17 points higher on average and performs better on 10 of the 15 benchmarks. The aggregate advantage therefore extends beyond mathematics, although the gains are not uniform across tasks. The task-level results show a different balance across capabilities rather than a uniform improvement. We separately evaluate instruction following on IFEval, where the CMA-based model scores 1.36 points higher than the UD-based model. The results are averaged over three repeated runs, which show the same overall trend. Further details are provided in Appendix E .

Taken together, the three experiments show that the difference between the two pretraining recipes remains after increasingly broad SFT. The CMA-based models retain stronger mathematics performance in both the GSM8K-based SFT and Math&Code SFT with replay settings. They also maintain a higher aggregate score after Tulu-3 mixed-domain SFT and perform better on instruction following. Because the task-level gains are uneven, these results do not imply that CMA improves every capability uniformly. This variation makes more controlled curriculum design an important direction for future work.

## 5 Related Works

### 5.1 Open-Recipe Language Models

The core science and engineering of large-scale pretraining remain difficult to study when leading systems are either closed or release weights without the data and training recipe needed to reconstruct the run [ Luo et al., 2025a ] . The term “open model” therefore covers substantially different release practices. Widely used model families such as Qwen [ Yang et al., 2024a , Yang et al., 2024b , Yang et al., 2025a , Qwen Team, 2026 ] , Gemma [ Rivière et al., 2024 , Gemma Team, 2025 , Gemma Team, 2026 ] , Llama [ Meta AI, 2024b ] , LFM [ Liquid AI, 2025 ] , and Falcon [ Zuo et al., 2025 ] publish model weights and enough implementation information for evaluation and deployment. However, their exact pretraining corpora, sample order, and complete training state are not released, so an independent group cannot reconstruct the original training run. We refer to these releases as open-weight . In contrast, a open-recipe release aims to expose the model weights, training data or a reconstructible data recipe, training and evaluation code, and sufficiently detailed configurations and intermediate artifacts to support scientific inspection and reproduction [ Groeneveld et al., 2024 , Walsh et al., 2025 ] . Such releases turn a final checkpoint into a reproducible account of how the model was built, making controlled study possible for resource-limited research communities.

Open-recipe projects provide the broader experimental stack needed to study pretraining itself. Pythia releases ordered data and dense intermediate checkpoints; OLMo and OLMoE release data, training and evaluation code, checkpoints, and logs; and projects such as SmolLM3, Yulan-Mini-2.4B, and Instella provide public data mixtures and reproducible training configurations [ Biderman et al., 2023 , Groeneveld et al., 2024 , Muennighoff et al., 2025 , Bakouch et al., 2025 , Yiwen et al., 2025 , Liu et al., 2025 ] . These resources allow other researchers to investigate learning dynamics, data selection, scaling, and optimization without relying on undocumented industrial pipelines. Open-recipe models can also be competitive: as shown in Tables 3 and 4 , models such as SmolLM3-3B, Yulan-Mini-2.4B, and Instella-3B match or surpass several open-weight models of comparable scale on parts of our evaluation, making this direction important beyond reproducibility alone.

This openness and capability have nevertheless often required substantial resources. Yulan-Mini-2.4B was trained on 48 A800 GPUs, SmolLM3 on 384 H100 GPUs, and the first stage of Instella-3B on 128 MI300X GPUs [ Yiwen et al., 2025 , Bakouch et al., 2025 , Liu et al., 2025 ] . Kaiyuan-2B, our previous work, provides a more resource-conscious example by improving data and training efficiency on Ascend 910A hardware [ Luo et al., 2025a ] . These examples expose a remaining gap between openness and practical accessibility: a training pipeline may be fully reproducible yet remain too costly for small research groups to rerun. Addressing this gap requires considering training workload, numerical precision, hardware accessibility, and explicit cost accounting together.

### 5.2 Low-Cost Language Model Pretraining

Training cost is meaningful only under an explicit accounting boundary. Token count and FLOPs alone capture only part of pretraining cost. End-to-end replication cost also depends on the workload required to reach a target quality, realized hardware throughput, and hardware price. We therefore consider cost jointly with model quality rather than using any single measure of compute or hardware efficiency as a proxy for low-cost training.