Puro-2B: Poor Lab's Qwen2-1.5B Trained on RTX 5090 within $5090 — part 1/7

Cleaned-text code points [0, 15588). Bibliography omitted here; source.md includes it.

\fancypagestyle firstpage \fancyhf \fancyfoot [C]1 \CJK@envStart UTF8rm \CJKtilde

# Puro-2B : Poor Lab’s Qwen2-1.5B Trained on RTX 5090 within $5090 Thanks: Corresponding authors.

###### 摘要

Language model pretraining has become almost synonymous with prohibitive cost, placing it out of reach for much of the academic and open-source communities. Although strong open-source efforts already exist, including open-weight models and open-source training recipes, a cost-efficient, hardware-accessible, and open-source pretraining recipe has long been missing. Even at a small scale, training Llama-3.2-3B costs over $1.5M, and reproducing SmolLM3-3B needs over $700K. In this report, we present an open pretraining recipe designed to lower this barrier. Using this recipe, we train a collection of Puro-2B (普罗-2B) models from scratch on up to 1.4 trillion tokens with FP8 precision on consumer-grade RTX 5090 GPUs 3 3 3 We sincerely thank Yanfu Investments for generously providing GPU computing resources to support this research. . The models in the collection differ in token budgets and selected recipe variants. Our best model is trained at a compute cost of less than $6.9K and approaches Qwen2.5-1.5B performance under our evaluation protocol. This cost efficiency is enabled by a combination of approaches, including hardware selection, low-precision training, hyperball optimization, curriculum model averaging, and the data recipe. Beyond the recipe itself, we provide two additional results. First, across the Puro-2B collection, we derive a Puro Cost Scaling Law that relates training cost to average model performance; the fitted law suggests that about $4.4K, less than $5,090 , is sufficient to reach the performance of Qwen2-1.5B. Second, as an end-to-end case study, we examine how pretraining data curricula shape downstream performance after post-training. Such controlled studies are enabled by having access to the full pretraining pipeline rather than model weights alone. We release the full training recipe for Puro-2B , including data, code, and model weights under Apache 2.0 at https://huggingface.co/collections/thu-pacman/puro-2b .

\fancyheadoffset 0.5cm \lhead Puro-2B : Poor Lab’s Qwen2-1.5B Trained on RTX 5090 within $5090 \rhead

## 1 Introduction

Scaling model parameters and training data has substantially improved the capabilities of large language models (LLMs) [ Hoffmann et al., 2022 , Kaplan et al., 2020 ] . However, the cost of pretraining limits the ability of academic and open-source researchers to study training behavior, reproduce complete pipelines, and test alternatives at meaningful scales. Lowering this barrier requires not only open model weights, but also reproducible data, software, infrastructure, training details, and transparent cost accounting.

Existing model releases span three levels of openness. Closed models , including proprietary families such as GPT, Claude, and Gemini [ OpenAI, 2023 , Gemini Team, 2025 ] , expose capabilities through hosted services and technical reports but do not release model weights or the artifacts required to inspect pretraining. Open-weight models , including the Qwen, Gemma, and Llama families [ Yang et al., 2024a , Yang et al., 2024b , Yang et al., 2025a , Rivière et al., 2024 , Gemma Team, 2025 , Meta AI, 2024b ] , release checkpoints that enable local evaluation, deployment, and post-training, but generally withhold the exact pretraining data, sample order, and complete training state. Open-recipe models go further by releasing reconstructible data mixtures, training code, configurations, and checkpoints. Projects such as OLMo and OLMoE, SmolLM, YuLan, Marin, and Instella therefore make the pretraining process itself available for scientific study [ Walsh et al., 2025 , Muennighoff et al., 2025 , Allal et al., 2025 , Bakouch et al., 2025 , Yiwen et al., 2025 , Liu et al., 2025 , Hall et al., 2025 ] .

Greater openness, however, does not by itself make pretraining affordable to reproduce. Even at a small scale, the compute cost of pretraining can be prohibitive. Under the rental-equivalent accounting in Figure 1 and Appendix B , training Llama3.2-3B costs over $1.5M in our estimation. For open-recipe models, reproducing OLMoE-1B-7B would cost $200K, and for SmolLM3-3B, the estimated reproduction cost rises to $719K. These cost budgets are derived from reported or estimated compute of these models. Therefore, even transparent open-recipe releases can remain open to the world but out of reach for many small and resource-constrained research labs — the poor labs , leaving a practical gap between reproducibility and accessibility.

To bridge this gap, we present a systematic reproducibility recipe covering system, algorithm, and data design for training a collection of Puro-2B (普罗-2B) checkpoints 4 4 4 Unless otherwise noted, we will use Puro-2B to denote the best one in this collection. . The recipe targets low-cost, accessible infrastructure for dense billion-parameter/trillion-token pretraining and produces a 2B model with performance matched against Qwen2-1.5B and Qwen2.5-1.5B under our protocol. These checkpoints share the same 2B-parameter architecture but differ in training budget and recipe variants. Training uses predominantly open-source datasets and runs on a cluster of consumer-grade RTX 5090 GPUs. Figure 1 compares counterpart models under a common cost-accounting protocol. Puro-2B attains competitive performance at the lowest reproduction cost among the plotted models under the stated accounting protocol, placing it beyond the baseline cost–performance frontier. Our best checkpoint surpasses Qwen2-1.5B overall and approaches Qwen2.5-1.5B. Detailed accounting is discussed in Section 4.3 and Appendix B As shown in Table 1 , the production schedule uses 438.8B Phase 1 tokens and 960.0B Phase 2 tokens. These stages total 1.4T scheduled tokens and 22,514 active-training GPU-hours, corresponding to a compute cost of about $6.9K and 17.6 elapsed days. The pipeline covers the data recipe, hardware infrastructure, supporting software, and training strategy.

We have released the following artifacts under the Apache 2.0 license unless otherwise noted separately; upstream data terms remain component-specific.

• Model weights: https://huggingface.co/thu-pacman/Puro-2B-Base and variants (10 different versions of checkpoints in total for better transparency and reproducibility)

• Data manifests and materialized components 5 5 5 Different licenses may apply to different dataset components, some of which may not be permissive. : https://huggingface.co/datasets/thu-pacman/Puro-2B

• Training code: https://github.com/thu-pacman/Puro-Megatron

• Data-processing code ( Kaiyuan-Spark ): https://github.com/thu-pacman/Kaiyuan-Spark

The recipe is built around five efficiency-oriented components spanning hardware, numerical precision, optimization, data ordering, and data selection. 1. We use consumer-grade RTX 5090 GPUs as an accessible training platform that provides a favorable cost-efficiency trade-off in our setting ( Section 3.1 ).

2. Blockwise FP8 training reduces per-token execution time while maintaining comparable model quality to bfloat16 (BF16) training ( Section 3.2 ).

3. We apply the MuonH optimizer , which extends the Muon optimizer with hyperball constraints on parameter weights and updates, together with a carefully designed learning rate schedule ( Section 3.3 ).

4. Curriculum Model Averaging (CMA) organizes training over coarse-grained data chunks according to configured source-local preferences and averages selected checkpoints [ Luo et al., 2026 ] ( Section 3.4 ).

5. Proxy experiments provide empirical signals for dataset selection and mixture design under a large candidate data pool ( Section 3.6 ).

Furthermore, we conduct targeted ablations over the first four design choices and report cost-efficiency estimates ( Figure 2 and Section 4.3 ). Our systematic recipe covers the full efficiency stack of pretraining: data selection reduces the required token budget, MuonH and the CMA recipe improve token efficiency by extracting more capability from training tokens, FP8 improves hardware utilization by increasing computational throughput, and hardware selection reduces the cost per unit of computation. Together, these choices optimize different bottlenecks across the training pipeline rather than targeting a single part.

Importantly, these components are not independent optimizations combined in isolation; they form a co-designed training system with interactions across different layers of the pipeline. For example, the Blackwell architecture of RTX 5090 provides the hardware foundation for efficient FP8 training. Both MuonH and CMA involve careful learning rate schedule design, and data curriculum relies on preprocessing decisions that determine the ordering of source-local data chunks. Therefore, the effectiveness of our recipe comes from the coordinated design of data, optimization, algorithmic, and system-level choices as an integrated approach to affordable pretraining.

In addition, we formulate the Puro Cost Scaling Law , a recipe-specific scale-down relationship between rental-equivalent training budget and model capability. As illustrated in Figure 2(b) , the law is designed for scale-down scenarios, serving users with limited compute and time budgets. There are three Phase 2 recipes used: U niform ordering with LR D ecay (UD), data C urriculum with LR D ecay (CD), and CMA. Their detailed ordering, learning rate schedule, and averaging choices are defined in Section 3.4 . The scale-down recipe uses the UD recipe. We resume training from the same Phase 1 checkpoints and adopt different Phase 2 token budgets. We then evaluate the resulting checkpoints and fit the trend of model performance as a function of total training cost. This scaling curve provides a reference for estimating the model capability achievable under a limited budget with our recipe. Interestingly, the Puro Cost Scaling Law suggests that our recipe can cross the Qwen2-1.5B performance aggregate at a cost of USD 4.4K. Its accounting assumptions and recipe-level interpretation are detailed in Section 4.3 . Moreover, the CD and CMA adoption can provide a further speedup compared to the UD scaling trend.

Beyond pretraining, the following post-training reveals that the difference between the CMA and UD Phase 2 recipes persists after matched supervised adaptation. In GSM8K-based SFT and Math&Code SFT with replay, CMA-based SFT yields higher GSM8K accuracy in repeated runs. In the Tulu-3 mixed-domain SFT setting, CMA-based SFT also improves the comprehensive capabilities and instruction following. Our Puro-2B recipes make these kinds of end-to-end comparisons possible at a meaningful scale and within an affordable budget.

In summary, our contributions are: 1. We provide a cost-effective, from-scratch pretraining recipe and train a Puro-2B model collection. The model in this collection exceeds Qwen2-1.5B at about $4.4K, and its best model checkpoint approaches Qwen2.5-1.5B with $6.9K. More broadly, our main contribution is to show that affordable, open pretraining is practical today ; we expect future efforts to push this frontier toward even lower cost and higher efficiency.

2. We evaluate the main recipe choices: RTX 5090 infrastructure, blockwise FP8, MuonH, and curriculum model averaging with supporting ablation experiments. Moreover, we formulate the Puro Cost Scaling Law for the recipe’s scale-down budget capability trend.

3. We release the datasets, model weights, intermediate checkpoints, training configurations, and implementation provided to reproduce and inspect this pretraining recipe, along with a post-training case study to show how this recipe helps end-to-end comparison for one pretraining design.

## 2 Overview

Before discussing individual design choices in detail, we first provide an overview of the complete training recipe. The pipeline starts with constructing the training corpus from publicly accessible sources and selecting cost-efficient, widely accessible hardware. It then proceeds through two phases of pretraining, including our optimization and data-curriculum designs, followed by model averaging, post-training, and evaluation. We also define the cost-accounting protocol and its boundary to illustrate the reported training costs. The following sections then discuss the motivation, implementation, and evidence for each major design choice. The overall pipeline is illustrated in Figure 3 .

### 2.1 Pipeline at a Glance

##### Model architecture and pretraining system ( Sections 3.1 and 3.2 ).

We train a dense decoder-only Transformer from scratch through two consecutive pretraining phases. The model adopts the Qwen3-1.7B architectural configuration, but unties the input embedding matrix from the output language-model head. This yields approximately 2B parameters in total. The Qwen3 architecture is familiar to the community and broadly supported. The model scale is also compatible with our replication budget. Our training is implemented in Megatron Core and adapted to an RTX 5090 cluster. To exploit the GPUs’ Blackwell FP8 support, the main Transformer linear-layer GEMMs use blockwise FP8, while numerically sensitive operations and persistent training states, including master weights and optimizer states, remain in BF16 or FP32 ( Section 3.2 ). Phase 1 runs on 24 GPUs, and Phase 2 continues from the Phase 1 endpoint on 96 GPUs 6 6 6 Yanfu investments help extend our compute resources in Phase 2. . The distributed parallel configuration is adjusted for the larger GPU count. The detailed low-precision compute flow, RTX 5090 system adaptations, and phase-specific parallel configurations are provided in Sections 3.1.1 and 3.1.3 .

##### Two-phase pretraining, curriculum, and optimization ( Sections 3.3 and 3.4 ).

Using the architecture and training system above, we pretrain Puro-2B in two phases. Phase 1 processes 438.8B tokens using the Phase 1 mixture, while Phase 2 consumes 960.0B tokens in total. The canonical run applies a data curriculum over the Phase 2 data pool ( Section 3.4 ), to present the more preferred portion of each scored source later in training. Rather than defining a global quality score across datasets, we primarily use the quality labels provided by source datasets. Within each dataset with usable score labels, examples are ordered from lower to higher score and partitioned into coarse chunks. Datasets without usable scores are instead randomly ordered and partitioned in the same way. Chunks from different sources are then aligned by their normalized within-source ranks, so that Phase 2 progresses from lower to higher configured ranks while approximately preserving the cross-component mixture. An illustration of this curriculum design is provided in Figure 7 .

To preserve the influence of data presented late in the curriculum, we jointly design the late-stage learning rate schedule and checkpoint averaging, following the curriculum-aware averaging principle of Luo et al. [2026] , and use the averaged weights as the final model. Phase 2 therefore has recipe variants: UD uses a uniform data ordering with LR decay, CD uses the component-local data curriculum with LR decay, and CMA adds the late constant-LR continuation and six-checkpoint average. For comparison, the UD variant uses the same Phase 1 checkpoint and Phase 2 pool but globally reshuffles the Phase 2 data instead of following the curriculum ordering.