\fancypagestyle firstpage \fancyhf \fancyfoot [C]1 \CJK@envStart UTF8rm \CJKtilde

# Puro-2B : Poor Lab’s Qwen2-1.5B Trained on RTX 5090 within $5090 Thanks: Corresponding authors.

###### 摘要

Language model pretraining has become almost synonymous with prohibitive cost, placing it out of reach for much of the academic and open-source communities. Although strong open-source efforts already exist, including open-weight models and open-source training recipes, a cost-efficient, hardware-accessible, and open-source pretraining recipe has long been missing. Even at a small scale, training Llama-3.2-3B costs over $1.5M, and reproducing SmolLM3-3B needs over $700K. In this report, we present an open pretraining recipe designed to lower this barrier. Using this recipe, we train a collection of Puro-2B (普罗-2B) models from scratch on up to 1.4 trillion tokens with FP8 precision on consumer-grade RTX 5090 GPUs 3 3 3 We sincerely thank Yanfu Investments for generously providing GPU computing resources to support this research. . The models in the collection differ in token budgets and selected recipe variants. Our best model is trained at a compute cost of less than $6.9K and approaches Qwen2.5-1.5B performance under our evaluation protocol. This cost efficiency is enabled by a combination of approaches, including hardware selection, low-precision training, hyperball optimization, curriculum model averaging, and the data recipe. Beyond the recipe itself, we provide two additional results. First, across the Puro-2B collection, we derive a Puro Cost Scaling Law that relates training cost to average model performance; the fitted law suggests that about $4.4K, less than $5,090 , is sufficient to reach the performance of Qwen2-1.5B. Second, as an end-to-end case study, we examine how pretraining data curricula shape downstream performance after post-training. Such controlled studies are enabled by having access to the full pretraining pipeline rather than model weights alone. We release the full training recipe for Puro-2B , including data, code, and model weights under Apache 2.0 at https://huggingface.co/collections/thu-pacman/puro-2b .

\fancyheadoffset 0.5cm \lhead Puro-2B : Poor Lab’s Qwen2-1.5B Trained on RTX 5090 within $5090 \rhead

## 1 Introduction

Scaling model parameters and training data has substantially improved the capabilities of large language models (LLMs) [ Hoffmann et al., 2022 , Kaplan et al., 2020 ] . However, the cost of pretraining limits the ability of academic and open-source researchers to study training behavior, reproduce complete pipelines, and test alternatives at meaningful scales. Lowering this barrier requires not only open model weights, but also reproducible data, software, infrastructure, training details, and transparent cost accounting.

Existing model releases span three levels of openness. Closed models , including proprietary families such as GPT, Claude, and Gemini [ OpenAI, 2023 , Gemini Team, 2025 ] , expose capabilities through hosted services and technical reports but do not release model weights or the artifacts required to inspect pretraining. Open-weight models , including the Qwen, Gemma, and Llama families [ Yang et al., 2024a , Yang et al., 2024b , Yang et al., 2025a , Rivière et al., 2024 , Gemma Team, 2025 , Meta AI, 2024b ] , release checkpoints that enable local evaluation, deployment, and post-training, but generally withhold the exact pretraining data, sample order, and complete training state. Open-recipe models go further by releasing reconstructible data mixtures, training code, configurations, and checkpoints. Projects such as OLMo and OLMoE, SmolLM, YuLan, Marin, and Instella therefore make the pretraining process itself available for scientific study [ Walsh et al., 2025 , Muennighoff et al., 2025 , Allal et al., 2025 , Bakouch et al., 2025 , Yiwen et al., 2025 , Liu et al., 2025 , Hall et al., 2025 ] .

Greater openness, however, does not by itself make pretraining affordable to reproduce. Even at a small scale, the compute cost of pretraining can be prohibitive. Under the rental-equivalent accounting in Figure 1 and Appendix B , training Llama3.2-3B costs over $1.5M in our estimation. For open-recipe models, reproducing OLMoE-1B-7B would cost $200K, and for SmolLM3-3B, the estimated reproduction cost rises to $719K. These cost budgets are derived from reported or estimated compute of these models. Therefore, even transparent open-recipe releases can remain open to the world but out of reach for many small and resource-constrained research labs — the poor labs , leaving a practical gap between reproducibility and accessibility.

To bridge this gap, we present a systematic reproducibility recipe covering system, algorithm, and data design for training a collection of Puro-2B (普罗-2B) checkpoints 4 4 4 Unless otherwise noted, we will use Puro-2B to denote the best one in this collection. . The recipe targets low-cost, accessible infrastructure for dense billion-parameter/trillion-token pretraining and produces a 2B model with performance matched against Qwen2-1.5B and Qwen2.5-1.5B under our protocol. These checkpoints share the same 2B-parameter architecture but differ in training budget and recipe variants. Training uses predominantly open-source datasets and runs on a cluster of consumer-grade RTX 5090 GPUs. Figure 1 compares counterpart models under a common cost-accounting protocol. Puro-2B attains competitive performance at the lowest reproduction cost among the plotted models under the stated accounting protocol, placing it beyond the baseline cost–performance frontier. Our best checkpoint surpasses Qwen2-1.5B overall and approaches Qwen2.5-1.5B. Detailed accounting is discussed in Section 4.3 and Appendix B As shown in Table 1 , the production schedule uses 438.8B Phase 1 tokens and 960.0B Phase 2 tokens. These stages total 1.4T scheduled tokens and 22,514 active-training GPU-hours, corresponding to a compute cost of about $6.9K and 17.6 elapsed days. The pipeline covers the data recipe, hardware infrastructure, supporting software, and training strategy.

We have released the following artifacts under the Apache 2.0 license unless otherwise noted separately; upstream data terms remain component-specific.

• Model weights: https://huggingface.co/thu-pacman/Puro-2B-Base and variants (10 different versions of checkpoints in total for better transparency and reproducibility)

• Data manifests and materialized components 5 5 5 Different licenses may apply to different dataset components, some of which may not be permissive. : https://huggingface.co/datasets/thu-pacman/Puro-2B

• Training code: https://github.com/thu-pacman/Puro-Megatron

• Data-processing code ( Kaiyuan-Spark ): https://github.com/thu-pacman/Kaiyuan-Spark

The recipe is built around five efficiency-oriented components spanning hardware, numerical precision, optimization, data ordering, and data selection. 1. We use consumer-grade RTX 5090 GPUs as an accessible training platform that provides a favorable cost-efficiency trade-off in our setting ( Section 3.1 ).

2. Blockwise FP8 training reduces per-token execution time while maintaining comparable model quality to bfloat16 (BF16) training ( Section 3.2 ).

3. We apply the MuonH optimizer , which extends the Muon optimizer with hyperball constraints on parameter weights and updates, together with a carefully designed learning rate schedule ( Section 3.3 ).

4. Curriculum Model Averaging (CMA) organizes training over coarse-grained data chunks according to configured source-local preferences and averages selected checkpoints [ Luo et al., 2026 ] ( Section 3.4 ).

5. Proxy experiments provide empirical signals for dataset selection and mixture design under a large candidate data pool ( Section 3.6 ).

Furthermore, we conduct targeted ablations over the first four design choices and report cost-efficiency estimates ( Figure 2 and Section 4.3 ). Our systematic recipe covers the full efficiency stack of pretraining: data selection reduces the required token budget, MuonH and the CMA recipe improve token efficiency by extracting more capability from training tokens, FP8 improves hardware utilization by increasing computational throughput, and hardware selection reduces the cost per unit of computation. Together, these choices optimize different bottlenecks across the training pipeline rather than targeting a single part.

Importantly, these components are not independent optimizations combined in isolation; they form a co-designed training system with interactions across different layers of the pipeline. For example, the Blackwell architecture of RTX 5090 provides the hardware foundation for efficient FP8 training. Both MuonH and CMA involve careful learning rate schedule design, and data curriculum relies on preprocessing decisions that determine the ordering of source-local data chunks. Therefore, the effectiveness of our recipe comes from the coordinated design of data, optimization, algorithmic, and system-level choices as an integrated approach to affordable pretraining.

In addition, we formulate the Puro Cost Scaling Law , a recipe-specific scale-down relationship between rental-equivalent training budget and model capability. As illustrated in Figure 2(b) , the law is designed for scale-down scenarios, serving users with limited compute and time budgets. There are three Phase 2 recipes used: U niform ordering with LR D ecay (UD), data C urriculum with LR D ecay (CD), and CMA. Their detailed ordering, learning rate schedule, and averaging choices are defined in Section 3.4 . The scale-down recipe uses the UD recipe. We resume training from the same Phase 1 checkpoints and adopt different Phase 2 token budgets. We then evaluate the resulting checkpoints and fit the trend of model performance as a function of total training cost. This scaling curve provides a reference for estimating the model capability achievable under a limited budget with our recipe. Interestingly, the Puro Cost Scaling Law suggests that our recipe can cross the Qwen2-1.5B performance aggregate at a cost of USD 4.4K. Its accounting assumptions and recipe-level interpretation are detailed in Section 4.3 . Moreover, the CD and CMA adoption can provide a further speedup compared to the UD scaling trend.

Beyond pretraining, the following post-training reveals that the difference between the CMA and UD Phase 2 recipes persists after matched supervised adaptation. In GSM8K-based SFT and Math&Code SFT with replay, CMA-based SFT yields higher GSM8K accuracy in repeated runs. In the Tulu-3 mixed-domain SFT setting, CMA-based SFT also improves the comprehensive capabilities and instruction following. Our Puro-2B recipes make these kinds of end-to-end comparisons possible at a meaningful scale and within an affordable budget.

In summary, our contributions are: 1. We provide a cost-effective, from-scratch pretraining recipe and train a Puro-2B model collection. The model in this collection exceeds Qwen2-1.5B at about $4.4K, and its best model checkpoint approaches Qwen2.5-1.5B with $6.9K. More broadly, our main contribution is to show that affordable, open pretraining is practical today ; we expect future efforts to push this frontier toward even lower cost and higher efficiency.

2. We evaluate the main recipe choices: RTX 5090 infrastructure, blockwise FP8, MuonH, and curriculum model averaging with supporting ablation experiments. Moreover, we formulate the Puro Cost Scaling Law for the recipe’s scale-down budget capability trend.

3. We release the datasets, model weights, intermediate checkpoints, training configurations, and implementation provided to reproduce and inspect this pretraining recipe, along with a post-training case study to show how this recipe helps end-to-end comparison for one pretraining design.

## 2 Overview

Before discussing individual design choices in detail, we first provide an overview of the complete training recipe. The pipeline starts with constructing the training corpus from publicly accessible sources and selecting cost-efficient, widely accessible hardware. It then proceeds through two phases of pretraining, including our optimization and data-curriculum designs, followed by model averaging, post-training, and evaluation. We also define the cost-accounting protocol and its boundary to illustrate the reported training costs. The following sections then discuss the motivation, implementation, and evidence for each major design choice. The overall pipeline is illustrated in Figure 3 .

### 2.1 Pipeline at a Glance

##### Model architecture and pretraining system ( Sections 3.1 and 3.2 ).

We train a dense decoder-only Transformer from scratch through two consecutive pretraining phases. The model adopts the Qwen3-1.7B architectural configuration, but unties the input embedding matrix from the output language-model head. This yields approximately 2B parameters in total. The Qwen3 architecture is familiar to the community and broadly supported. The model scale is also compatible with our replication budget. Our training is implemented in Megatron Core and adapted to an RTX 5090 cluster. To exploit the GPUs’ Blackwell FP8 support, the main Transformer linear-layer GEMMs use blockwise FP8, while numerically sensitive operations and persistent training states, including master weights and optimizer states, remain in BF16 or FP32 ( Section 3.2 ). Phase 1 runs on 24 GPUs, and Phase 2 continues from the Phase 1 endpoint on 96 GPUs 6 6 6 Yanfu investments help extend our compute resources in Phase 2. . The distributed parallel configuration is adjusted for the larger GPU count. The detailed low-precision compute flow, RTX 5090 system adaptations, and phase-specific parallel configurations are provided in Sections 3.1.1 and 3.1.3 .

##### Two-phase pretraining, curriculum, and optimization ( Sections 3.3 and 3.4 ).

Using the architecture and training system above, we pretrain Puro-2B in two phases. Phase 1 processes 438.8B tokens using the Phase 1 mixture, while Phase 2 consumes 960.0B tokens in total. The canonical run applies a data curriculum over the Phase 2 data pool ( Section 3.4 ), to present the more preferred portion of each scored source later in training. Rather than defining a global quality score across datasets, we primarily use the quality labels provided by source datasets. Within each dataset with usable score labels, examples are ordered from lower to higher score and partitioned into coarse chunks. Datasets without usable scores are instead randomly ordered and partitioned in the same way. Chunks from different sources are then aligned by their normalized within-source ranks, so that Phase 2 progresses from lower to higher configured ranks while approximately preserving the cross-component mixture. An illustration of this curriculum design is provided in Figure 7 .

To preserve the influence of data presented late in the curriculum, we jointly design the late-stage learning rate schedule and checkpoint averaging, following the curriculum-aware averaging principle of Luo et al. [2026] , and use the averaged weights as the final model. Phase 2 therefore has recipe variants: UD uses a uniform data ordering with LR decay, CD uses the component-local data curriculum with LR decay, and CMA adds the late constant-LR continuation and six-checkpoint average. For comparison, the UD variant uses the same Phase 1 checkpoint and Phase 2 pool but globally reshuffles the Phase 2 data instead of following the curriculum ordering.

Across both phases, we use Hyperball optimization for selected approximately scale-invariant matrices. MuonH updates these matrices and projects each one back to its initial Frobenius radius after every step, while AdamW updates the remaining parameters ( Section 3.3 ). Both parameter groups share a base LR schedule. MuonH group uses a 10 times base LR schedule as Hyperball weight LR. The base schedule follows a power schedule [ Shen et al., 2024b ] in Phase 1 and continues from its terminal Phase 1 value with linear decay along the main Phase 2 trajectory. The late CMA continuation instead holds the base LR fixed at its resume-point value before checkpoint averaging ( Section 3.4 ). Both phases use a sequence length of 4,096 tokens and a global batch size of 1,536 sequences. The optimizer configuration, learning rate multipliers, curriculum construction, transition schedule, checkpoint-averaging rule, and exact token accounting are reported in Sections 3.3 and 3.4 , Appendix C , and Table 7 .

##### Data acquisition, filtering, and mixture ( Section 3.6 ).

We construct the training corpus from publicly accessible datasets with licenses or terms that permit their use, enabling the corpus to be reproducibly reconstructed from the original sources. The corpus spans web text, code, and mathematics, and also includes synthetic examples. We first use our Kaiyuan-Spark framework ( Section 3.6.2 ) to deduplicate the large-scale web portion. Because the available data far exceeds our training budget, we must select source datasets, filter samples within each source, and combine the retained portions to improve overall data utility and match the desired capability profile. However, it is difficult to design a single sample-level quality score that is comparable across domains, apply it to every sample, and use it to determine which samples to retain. Instead, we run proxy experiments on candidate sources and representative data slices ( Section 3.6.1 ). Specifically, we train controlled small-scale models on these sources or slices and evaluate the resulting models on downstream tasks. The resulting vector of benchmark scores defines the proxy profile of each source or slice, which guides source selection and mixture design ( Section 3.6.1 ). For example, when stronger mathematical capability is desired, we can assign higher weights to datasets whose proxy profiles show stronger performance on mathematics benchmarks. Some open-source datasets also provide sample-level quality scores. Although these scores are not directly comparable across sources, they can support within-source filtering. For each such dataset, we sample proxy slices from different score ranges, compare their proxy profiles, and use the results to select a dataset-specific threshold for retaining high-scoring samples. The final domain-level recipe for the two pretraining phases is summarized in Figure 8 and Figure 8 , while reports family-level retained token counts, selection modes, assigned phases, source identifiers, and license terms rather than a complete component-level manifest.

##### Evaluation and post-training ( Sections 3.5 and 4 ).

Our evaluation combines a broad pretrained-model comparison with controlled tests of downstream adaptation. We first evaluate Puro-2B against open-weight and reproducible open-recipe models of comparable scale on general reasoning, mathematics, and coding. Under the report’s common budget accounting, Puro-2B surpasses Qwen2-1.5B and Gemma-2-2B while using less than one sixth of the stated training cost of comparable open recipes. We then use SFT as a probe of how the Phase 2 recipes transfer through post-training. Within each paired experiment, the CMA and UD initializations receive identical supervised data and optimization; mathematics checkpoints use one frozen generation and extraction protocol, while broad tasks use benchmark-appropriate prompts and scorers. The three settings are GSM8K-based SFT, Math&Code SFT with replay, and Tulu-3 mixed-domain SFT, with complete definitions and results in Section 3.5 and Appendix E .

### 2.2 Reproduction-Cost Boundary

Throughout this report, reproduction cost refers specifically to the compute cost of rerunning the finalized two-phase pretraining recipe once. The accounting boundary begins after the phase-specific training shards have been materialized and includes only the compute time of the Phase 1 and Phase 2 production runs. We report measured GPU-hours separately and convert them using the normalized RTX 5090 rental-equivalent rate specified in Section 4.3 and Appendix B . The ledger retains CNY(RMB) for traceability, while the main cost figures use the corresponding USD values. Both are reproducible cost proxies under the stated rate, not the authors’ total cash expenditure.

We do not estimate costs outside these production pretraining runs. The headline therefore excludes data acquisition and preprocessing, proxy-model experiments, scaling and ablation studies, failed or exploratory runs, post-training, evaluation, checkpoint averaging, and research labor. It also excludes non-accelerator resources such as CPU processing, storage, and networking, as well as taxes, depreciation, and other ownership costs not represented by the rental-equivalent rate. These exclusions do not imply that the activities are free; they mean that the reported number should be interpreted as a narrowly defined marginal accelerator cost for reproducing the final pretraining run, rather than as the total cost of developing the model or reconstructing every artifact from scratch.

## 3 Training Recipe

We organize the training recipe as follows. We first describe the infrastructure design that supports our training setup. We then present the pretraining recipe, focusing on optimization and data curriculum. Next, we explain how the training dataset is constructed. Finally, we present a post-training case study to demonstrate how our recipe enables end-to-end approach comparison.

### 3.1 Training Infrastructure

#### 3.1.1 RTX 5090 as a Cost-Effective GPU Choice

To make our recipe cost-efficient and the hardware accessible to the broader community, we made a non-standard infrastructure choice: we run all pretraining and post-training on a cluster equipped with consumer-grade GPUs (RTX 5090), instead of renting conventional data-center-oriented accelerators. Although data-center GPUs are generally expected to be more stable and compute-efficient, our cost analysis showed that RTX 5090 is a better fit for this workload.

{narrowtalltblr} [ caption = Peak hardware capability and cost efficiency of candidate NVIDIA GPUs., label = tab:gpu-cost-efficiency, notea = All performance numbers are from official NVIDIA documentation / datasheets / blog posts Andersch et al. [2022] , Corporation [2024] , Corporation [2025a] , Corporation [2025b] and refer to dense matrix operations with FP32 accumulation., noteb = The prices do not include CPU / storage / network, as they are marginal and often free on GPU rental platforms. A100, H200, and RTX PRO 6000 prices were collected from a public GPU rental platform on 16 Aug 2026. RTX 5090 has no widely available public rental channel due to NVIDIA EULA restrictions; we estimate its effective price by amortizing our hardware and electricity costs over five years. An 8-GPU node costs approximately 12,000 CNY ($1763) per month for our setup. Some private compute providers confirmed similar rental pricing for RTX 5090., notec = Hopper FP8 Tensor Cores use a reduced-precision path for nominal FP32 accumulation, with 22 effective precision bits Zhang et al. [2025a] , which may affect numerical accuracy., noted = PCIe Gen5 × \times 16 provides approximately 64 GB/s of theoretical bandwidth per direction, or approximately 128 GB/s of aggregate bidirectional bandwidth. Driver tweaks needed to achieve this bandwidth are discussed in Section 3.1.2 ., ] colspec = l r r r c c c c c r r r, rowhead = 2, row1-2 = font= , bg=gray!10 , cell11 = r=2l, cell12 = c=3c, cell15 = r=2c, cell16 = r=2c, cell17 = r=2c, cell18 = r=2c, cell19 = r=2c, cell110 = c=3c, hline3 = 0.5pt, GPU Model Peak tensor performance (TFLOPS) \TblrNote a Memory capacity Memory bandwidth TDP Intra-node P2P BW (Bi-Di) Single-GPU price \TblrNote b Peak compute/$ (EFLOP/USD) BF16 FP8 NVFP4 BF16 FP8 FP4 A100 SXM4 312 N/A N/A 80 GB 2 TB/s 400 W 600 GB/s $1.79/h 0.63 N/A N/A H200 SXM5 989.5 1979 \TblrNote c N/A 141 GB 4.8 TB/s 700 W 900 GB/s $4.00/h 0.89 1.78 N/A RTX PRO 6000 503.8 1007.6 2015.2 96 GB 1.8 TB/s 600 W 128 GB/s \TblrNote d $1.89/h 0.96 1.92 3.85 RTX 5090 209.5 419 1676 32 GB 1.8 TB/s 575 W 128 GB/s \TblrNote d $0.31/h 2.43 4.87 19.46

shows the features and prices of different generations of NVIDIA GPUs. RTX 5090 has a clear gap in absolute peak throughput compared with data-center GPUs, especially because its Tensor Cores are artificially limited to half of actual peak performance when using FP32 accumulation. However, its much lower effective price gives substantially higher compute per dollar: its BF16 and FP8 cost efficiency is about 2.7 × \times that of H200. Moreover, because policy and cost constraints made Blackwell data-center accelerators such as GB200 unavailable to us, RTX 5090 is also the most practical option with FP4 support.

RTX 5090 still has significant hardware limitations, including smaller memory capacity and the lack of NVLink for high-bandwidth GPU-to-GPU scaling. These drawbacks cannot be ignored. Nevertheless, our target model is small enough that, with the hardware configuration and parallel training setup described below, the RTX 5090 cluster reaches a mixed-precision effective MFU of approximately 73% under the stated precision-weighted peak convention. Section 3.1.3 defines the convention and measurement boundary; this is not a BF16-only full-scale result. Its cost advantage therefore remains valid for our setting.

#### 3.1.2 Hardware Setup and Tweaks

Our training cluster consists of multiple x86 GPU servers. Each server is equipped with dual CPUs, eight RTX 5090 GPUs (four in each CPU socket), and sufficient host memory. The performance numbers in Section 3.1.1 are single-GPU metrics; for multi-GPU training, both intra-node P2P bandwidth and inter-node network bandwidth must be considered to sustain training throughput.

##### Intra-node bandwidth.

Since RTX 5090 does not provide NVLink, the only intra-node interconnect is PCIe. NVIDIA disables PCIe P2P on non-data-center GPUs, which forces GPU communication traffic to be staged through host memory, also known as “ping-pong” transfer, which can substantially reduce communication efficiency. However, this restriction appears to be enforced by the NVIDIA GPU driver rather than the limitation of GPU hardware. We used a modified version of the open-source NVIDIA driver aikitoria [2026] , along with the necessary platform configuration changes Chen [2026b] , including disabling IOMMU and PCIe ACS as well as adjusting NPS (NUMA Per Socket, also known as Sub-NUMA Clustering on some platforms) to enable GPU P2P on our servers.

On our hardware, enabling P2P improved one-way bandwidth from 31.5 GB/s to 56 GB/s. The former is bounded by half of the PCIe 5.0 × \times 16 bandwidth, i.e., 32 GB/s, because ping-pong transfer requires two host-memory copies; the latter approaches the 64 GB/s PCIe link limit. Bidirectional bandwidth increased from 32 GB/s to 111 GB/s, compared with theoretical limits of 128 GB/s. Communication latency decreased drastically from 14.3 μ \mu s to 0.4 μ \mu s. The eight-GPU AllReduce bandwidth over PCIe, measured as busbw reported by nccl-tests , increased from 14.75 GB/s to 27.34 GB/s. When the test was restricted to the four GPUs attached to a single CPU socket, the gain was larger, from 16.33 GB/s to 46.31 GB/s, about 2.8 × \times . Adding a PLX PCIe switch between the GPUs and the root complex might further improve P2P bandwidth toward the link limit; this is a prospective topology change, not a configuration used in the reported run.

##### Inter-node bandwidth.

To scale training beyond a single server, the inter-node network must provide bandwidth comparable to the intra-node communication path. We set up a 400 Gbps InfiniBand network among our servers, i.e., 100 GB/s bidirectional bandwidth: one dual-port Mellanox ConnectX-7 host card adapter (HCA) is installed in each server, with both ports connected to a 200 Gbps InfiniBand HDR switch. Half of the GPUs in the server could communicate with the HCA directly, while the other half must route through the inter-socket link to reach the HCA.

Even after PCIe P2P is enabled, non-data-center NVIDIA GPUs still do not support GPUDirect RDMA (shortened as GDR), another performance-critical feature for inter-node communication. GDR builds on P2P and allows a GPU to access remote GPU memory over RDMA NICs without staging through host memory. Through testing, we found that this restriction is also software-enforced: a minimal binary modification to the CUDA user-space driver can enable GDR on RTX 5090 Chen [2026a] . Due to EULA constraints, we are not able to disclose the details in this report. When this configuration is unavailable, the same training recipe can be run with the stock NVIDIA driver, falling back to the standard communication path at lower inter-node bandwidth. In end-to-end tests, enabling GDR improved the 24-GPU (3 servers) AllReduce bandwidth, measured as busbw , from approximately 8.87 GB/s to 19.93 GB/s. We believe that if a more balanced network topology is used (e.g., one InfiniBand HCA on each socket), the performance gain could be even higher.

##### Caveats.

These modifications should be used with caution. First, modifying drivers is unsupported by the vendor and must be done at the user’s own risk. Second, these changes cannot exceed the physical PCIe bandwidth limit. Third, P2P and GDR should not be enabled blindly; the optimal setting depends on the actual hardware topology and platform capability. For example, if the PCIe root complex of a CPU has limited peer-switching capability (which is often the case on low-end models), heavy P2P traffic among many devices may become congested, and host-memory ping-pong transfer can be faster.

#### 3.1.3 Efficient Training System

We build our training system on Megatron Core Shoeybi et al. [2019] , specifically release v0.16.0 NVIDIA Corporation [2026a] , together with its Transformer Engine dependency. This software stack provides native support for blockwise FP8 training and the Muon optimizer, making it a suitable basis for a model at our scale. Nevertheless, the combination of consumer GPUs, a mixed-precision training recipe, and our model architecture still requires careful system-level tuning.

##### Communication-aware parallelism.

The limited communication bandwidth of RTX 5090 motivates us to use only parallel dimensions with modest communication demands. We combine data parallelism (DP), whose dominant collectives are gradient ReduceScatter and parameter AllGather around optimization, with pipeline parallelism (PP), which exchanges hidden-state tensors only at stage boundaries Narayanan et al. [2021] . We do not use tensor parallelism (TP), which introduces frequent collectives within every Transformer layer. Compared with data-center-class systems, our communication disadvantage is substantially more pronounced within a node. We therefore depart from Megatron Core’s default rank ordering and use a pp-dp ordering that places each PP group on topologically closer GPUs within a node, thereby reducing sensitivity to communication latency.

##### Appropriate micro-batch size.

FP8 kernels complete their arithmetic more quickly while introducing additional quantization operations, so they become memory-bound more readily than BF16 kernels. This effect is amplified by the relatively small hidden dimension of our model. Increasing the micro-batch size (MBS) improves the arithmetic intensity and Tensor Core utilization, but an excessive MBS exhausts device memory: DP does not partition activations; and 1F1B pipeline schedule Narayanan et al. [2021] helps control the number of simultaneously resident microbatches, but it does not eliminate the per-microbatch activation-memory growth caused by increasing MBS. We therefore extend Megatron-LM’s analytical FLOP estimator to report the invocation shapes of GEMM and FlashAttention kernels. For each shape induced by our model configuration, we benchmark the corresponding Transformer Engine kernel and select the smallest MBS beyond which achieved FLOP/s no longer increases materially, analogous to locating the knee of a roofline curve.

##### Overcoming load imbalance.

Although our model is relatively small, its vocabulary is large; consequently, the embedding and LM head account for a non-negligible fraction of both computation and memory. FP8 GEMMs accelerate the internal Transformer layers, further increasing their performance gap with the LM head. As a result, the LM head alone entails computation comparable to several Transformer layers, creating substantial pipeline imbalance. Guided by our extended FLOP estimator, which attributes theoretical work by layer, submodule, operator, and precision, we assign fewer Transformer layers to the stage containing the LM head. This balances pipeline computation at the cost of greater memory pressure on the first stage. Tensors assigned to Muon are preferably kept intact rather than sharded. We replace the zigzag placement policy of the layer-wise distributed optimizer in this version of Megatron Core, which over-concentrates the embedding and LM-head tensors, with a memory-aware allocation DeepSeek-AI et al. [2026] : Muon tensors are greedily assigned by memory footprint like bin packing, after which flattened Adam tensors fill the remaining per-device capacity as evenly as possible. This hybrid placement balances optimizer memory across GPUs and is important for fitting the training state into the 32 GB memory available on each RTX 5090.

These principles yield the same fastest configuration as an exhaustive enumeration of the candidate parallel strategies. Together with the hardware optimizations described above and standard Megatron Core options such as the [T,H,D] layout for packed-sequence attention, the Phase 1 production run sustains a median 238 TFLOP/s per GPU with global batch size 1536 on 24 GPUs across three nodes. The best configuration is MBS = 2 {}=2 , PP = 2 {}=2 with layout (18|10) 7 7 7 The PP layout lists the numbers of Transformer layers assigned to successive stages. In (18|10) , the first stage contains the embedding and the first 18 Transformer layers, while the second contains the remaining 10 layers and the LM head. , and DP = 12 {}=12 . The measured throughput corresponds to an equivalent MFU 8 8 8 Model FLOPs utilization (MFU) is conventionally defined as achieved model FLOP/s divided by the accelerator’s peak FLOP/s Korthikanti et al. [2023] . For mixed-precision training, we weight the peak Tensor Core performance by the fraction of theoretical work at each precision. In our recipe, FP8 accounts for 72% of the theoretical Tensor Core operations and BF16 for the remaining 28%, giving an effective peak of P eff = ( 0.72 / 419 + 0.28 / 209.5 ) − 1 ≈ 327 P_{\mathrm{eff}}=(0.72/419+0.28/209.5)^{-1}\approx 327 TFLOP/s per GPU. of approximately 73%. For larger-scale Phase 2 training, strong scaling is limited by Amdahl’s law, and gradient synchronization accounts for a larger fraction of the step time. We consequently use PP = 4 {}=4 with layout (9|9|9|1) and DP = 24 {}=24 on 96 GPUs, while still achieving median throughput 192 TFLOP/s per GPU. The run-level setup and final losses are reported in Table 7 .

### 3.2 FP8 Mixed-Precision Training

We use FP8 mixed precision from random initialization onward, without a BF16 warm-up stage or a later precision switch, using our training stack NVIDIA Corporation [2026a] and its blockwise FP8 support NVIDIA Corporation [2026b] . The training state and numerically sensitive operations retain the standard BF16/FP32 mixed-precision path, while Transformer linear layers use E4M3 operands. FP8 is therefore an online compute and activation-storage format rather than the persistent dtype of the whole model.

E4M3 Micikevicius et al. [2022] provides more mantissa bits than E5M2 but has a narrower dynamic range. With one scale for an entire tensor, a few outliers can enlarge the quantization interval and reduce the effective precision of ordinary values DeepSeek-AI et al. [2024] . We instead compute scales online for local groups: activations and activation gradients use one-dimensional groups of 128 consecutive values along the GEMM reduction dimension, while weights use two-dimensional 128 × 128 128\times 128 blocks Yan et al. [2026] . This fine-grained numerical design follows the recipe introduced and validated at scale by DeepSeek-V3 DeepSeek-AI et al. [2024] .

On our RTX 5090 (SM 120) Corporation [2025a] , each logical block scale is constrained to a power of two and therefore has the exponent-only expressiveness of E8M0. Transformer Engine NVIDIA Corporation [2026b] implements these blocks through Blackwell’s native MXFP8 path. Thus, our logical block sizes follow DeepSeek-V3 DeepSeek-AI et al. [2024] , while the scaling-factor representation and execution path follow MXFP8; the fact that DeepSeek-V3 was trained on Hopper GPUs may account for this implementation difference.

The complete precision flow is shown in Figure 14 . Here, Transformer linear layers include the QKV projections, attention output projection, and MLP projections; their Fprop, Dgrad, and Wgrad GEMMs all use the FP8 path DeepSeek-AI et al. [2024] . Core attention refers only to the FlashAttention/SDPA computation between these projections and remains in BF16 Yan et al. [2026] . A linear layer quantizes its BF16 inputs and weights immediately before GEMM and materializes a BF16 output for the surrounding model. Its quantized input is retained for Wgrad, reducing the dominant saved-activation footprint Yan et al. [2026] . FP8-accelerated operations account for 72% of the overall computation; the cost impact and net benefit of FP8 are analyzed in Figure 2 .

The model-weight portion of each training checkpoint consequently remains BF16; the desired release or deployment representation is produced by a separate post-training conversion. In the matched 20-token-per-parameter scaling ladder, blockwise FP8 increases validation loss by 0.0031–0.0039 relative to BF16 across the five tested model sizes. The shared-shape fit maps this difference to 98.0% BF16-equivalent compute retention. At the 1.7B scale, however, blockwise FP8 improves median training throughput by 1.36 × 1.36\times , yielding a quality-adjusted net gain of 1.34 × 1.34\times . The complete setup, fitting procedure, and size-wise results are reported in Section 4.3.1 .

### 3.3 Hyperball Optimization

In our recipe, we adopt Muon with the Hyperball wrapper (MuonH), introduced by Wen et al. [2026] , as the training optimizer for selected weight matrices. We apply MuonH to approximately scale-invariant weights, including attention and MLP matrices, while embeddings, normalization layers, the language-model head, and the remaining parameters use AdamW. Following Wen et al. [2026] , MuonH normalizes the Muon update and constrains each wrapped matrix to a fixed-radius sphere. More specifically, for each wrapped matrix W t W_{t} , MuonH fixes the radius at R = ∥ W 0 ∥ F R=\lVert W_{0}\rVert_{F} and normalizes the Muon update u t u_{t} as u ^ t = u t / ∥ u t ∥ F \widehat{u}_{t}=u_{t}/\lVert u_{t}\rVert_{F} . It then applies W ~ t + 1 = W t − η t ​ R ​ u ^ t , W t + 1 = R ​ Normalize ⁡ ( W ~ t + 1 ) , \widetilde{W}_{t+1}=W_{t}-\eta_{t}R\widehat{u}_{t},\qquad W_{t+1}=R\,\operatorname{Normalize}(\widetilde{W}_{t+1}), (1) where Normalize ⁡ ( A ) = A / ∥ A ∥ F \operatorname{Normalize}(A)=A/\lVert A\rVert_{F} . The first operation normalizes the update scale, while the second projects the updated weight matrix back to its initial Frobenius radius. The illustration of MuonH optimizer is shown in Figure 4 .

We vary the learning rate η t \eta_{t} according to a two-phase learning rate schedule. The design of this schedule is motivated by an analysis of how the loss gap between MuonH and Muon arises from the perspective of effective learning rate, as we elaborate below.

#### 3.3.1 Effective Learning Rate

First, we briefly revisit the rationale behind Hyperball optimization and explain how it makes the effective LR schedule explicit [ Wen et al., 2026 ] . We then examine the loss gap between ordinary Muon and MuonH runs and find that the effective LR schedule strongly affects the loss convergence rate. This observation leads us to focus on learning rate schedule design for the MuonH optimizer.

##### Scale invariance motivates the effective learning rate.

Hyperball is motivated by approximately scale-invariant parameter matrices. For a selected weight matrix W W , we call the parameter approximately scale invariant when positively rescaling it has little effect on the model function or loss, ℒ ⁡ ( c ​ W ) ≈ ℒ ⁡ ( W ) , c > 0 , \mathcal{L}(cW)\approx\mathcal{L}(W),\qquad c>0, (2) with the remaining parameters held fixed [ Wen et al., 2026 ] . Normalization in modern Transformers makes this approximation relevant for several attention and MLP weight matrices. In this regime, the update magnitude is naturally interpreted relative to the current weight scale. We therefore consider the matrix-wise effective LR, also called the effective learning rate (ELR). For ordinary Muon, with Muon update u t u_{t} , scalar learning rate η t \eta_{t} , and decoupled weight-decay coefficient λ \lambda , W t + 1 = ( 1 − λ ​ η t ) ​ W t − η t ​ u t , W_{t+1}=\bigl(1-\lambda\eta_{t}\bigr)W_{t}-\eta_{t}u_{t}, (3) and, ignoring the radial shrinkage from weight decay, the effective LR is ρ t ​ ( W ) = η t ​ ∥ u t ∥ F ∥ W t ∥ F . \rho_{t}(W)=\frac{\eta_{t}\lVert u_{t}\rVert_{F}}{\lVert W_{t}\rVert_{F}}. (4) Thus, ordinary Muon’s effective LR is not determined by the scalar learning rate alone, because both the weight norm and the Muon-update norm evolve during training.

##### MuonH makes the effective LR schedule explicit.

Under the Hyperball update in Equation 1 , the pre-projection displacement η t ​ R ​ u ^ t \eta_{t}R\widehat{u}_{t} has norm η t ​ R \eta_{t}R , while the wrapped matrix W t W_{t} has norm R R . Its effective LR is therefore ρ t ​ ( W ) = η t ​ ∥ u t ∥ F / ∥ W t ∥ F = η t ​ R / R = η t \rho_{t}(W)=\eta_{t}\lVert u_{t}\rVert_{F}/\lVert W_{t}\rVert_{F}=\eta_{t}R/R=\eta_{t} , numerically equal to the Hyperball weight learning rate, and the subsequent projection restores the fixed radius. Thus, ordinary Muon induces its effective LR indirectly through the scalar learning rate and evolving norms, whereas MuonH directly controls the effective LR schedule through its learning rate schedule. This explicit control is the property of Hyperball that is most relevant to our subsequent schedule design.

In our experiments, we use different learning rates for the MuonH-wrapped and AdamW-optimized parameter groups. We define a base learning rate schedule η t base \eta_{t}^{\mathrm{base}} for the AdamW parameter groups and set the Hyperball weight learning rate to a fixed multiple, η t H = m ​ η t base . \eta_{t}^{H}=m\,\eta_{t}^{\mathrm{base}}. For MuonH-wrapped matrices, η t H \eta_{t}^{H} is numerically equal to the prescribed effective LR. In the production run, we use m = 10 m=10 , while the remaining parameter groups follow η t base \eta_{t}^{\mathrm{base}} directly with AdamW.

##### Matching the effective LR schedule also aligns validation loss.

The MuonH optimizer can make training more efficient than the ordinary Muon optimizer, and we find that the efficiency gap can be attributed to the difference in their effective LR schedule. We compare three 170M-parameter BF16 runs with the same architecture, batch size, sequence length, seed, and evaluation protocol. The MuonH run uses a predefined effective LR schedule ρ t H \rho_{t}^{H} . The ordinary-LR run uses the base LR schedule of the MuonH run as the LR schedule and does not compensate for the evolving weight and update norms. Then, as shown in Figure 5 , the ordinary-LR Muon run produces an effective LR trace that decays rapidly early in training and finishes close to zero. The MuonH run instead follows the predefined linear-decay effective-LR schedule. Although the ordinary-LR Muon run achieves lower validation loss early in training, MuonH overtakes it near the end. The MuonH run finishes at a validation loss of 3.029 3.029 , compared with 3.073 3.073 for ordinary-LR Muon.

For comparison, we conduct an effective-LR aligned run, which uses the ordinary Muon optimizer but adjusts its scalar learning rate online to follow the same effective LR curve, η t = ρ t H ​ ∥ W t ∥ F ∥ u t ∥ F , \eta_{t}=\rho_{t}^{H}\frac{\lVert W_{t}\rVert_{F}}{\lVert u_{t}\rVert_{F}}, (5) so that the effective LR ρ t ​ ( W ) = ρ t H \rho_{t}(W)=\rho_{t}^{H} without projecting W t + 1 W_{t+1} back to a fixed-radius sphere. As shown in Figure 5 , effective-LR aligned Muon follows the prescribed linear post-warmup LR decay and reaches a final validation loss of 3.030 3.030 , close to MuonH at 3.029 3.029 . Thus, in this selected diagnostic, ordinary Muon largely recovers the MuonH loss curve when its effective LR schedule is aligned . This comparison supports the effective-LR schedule as an important factor in MuonH’s training behavior. It also suggests that better control of the effective-LR schedule can improve loss convergence. Recent work also discusses related behavior from the perspective of angular update size [ Zhou et al., 2026b , Xiao et al., 2026 ] . We provide an additional ordinary Muon control in Section F.3 that can even use a counterintuitive hill-like scalar learning rate schedule to achieve loss convergence while maintaining the prescribed effective LR trajectory.

##### Effective LR provides a more informative descriptor of the loss trend.

The experiments above show a close relationship between effective LR and loss dynamics. We further test whether effective LR provides a more informative descriptor than the raw optimizer learning rate. We apply the Multi-Power Law (MPL), a LR-schedule scaling law that predicts loss-curve shape from the LR schedule, to both the effective-LR schedule and the raw learning rate schedule [ Luo et al., 2025b ] . For the two ordinary-Muon runs above, we fit the post-warmup validation curves using either the scalar learning rate or the induced effective LR, while holding out the final 20 % 20\% of validation targets. As shown in Figure 16 , using the effective LR reduces the mean held-out RMSE across the two runs from 0.0265 0.0265 to 0.0210 0.0210 . The improvement is driven mainly by the base-Muon run; for effective-LR aligned Muon, the two LR signals are comparably predictive. This small diagnostic therefore supports effective LR as a more informative descriptor for schedule-dependent training dynamics. The MPL formulation and fitting results are provided in Section F.1 .

##### Effective LR MPL helps interpret the late-stage crossover.

In the selected 170M diagnostic, MuonH and effective-LR aligned Muon converge more slowly at the beginning but outperform ordinary Muon near the end of training. This pattern is consistent with the learning rate analysis in Appendix F . In particular, the MPL fit [ Luo et al., 2025b ] provides a simple diagnostic relationship: a decrease in learning rate can induce an approximately proportional reduction in loss, with the loss decrease taking effect over subsequent training steps. Applying this view to the effective LR, ordinary Muon decays its effective LR more aggressively early in training, which is consistent with its faster early loss reduction but leaves less effective LR decay for the later stage. MuonH and aligned Muon instead follow a more gradual, approximately linear effective LR decay, giving up some early convergence speed while retaining more decay toward the end of training. This view provides additional motivation for using a linear-decay schedule in our production design.

Together, these diagnostics motivate treating effective LR as the primary hyperparameter for MuonH-wrapped weights. They also suggest that the placement of its decay can affect where loss reduction occurs during training. We therefore next study the practical design of the LR schedule for production run.

#### 3.3.2 Learning Rate Schedule Design

##### Continual training motivates an open-ended schedule followed by a controlled terminal decay.

In practice, the total token budget is commonly undetermined in the beginning [ Hu et al., 2024 , Shen et al., 2024b , Allal et al., 2025 ] . Hence, continual pretraining is a practical setting and may change the data mixture after the initial horizon. We therefore want to retain an open-ended schedule before applying a controlled terminal decay. Warmup-Stable-Decay (WSD) schedule provides a testbed for choosing the decay duration [ Hu et al., 2024 ] . Motivated by the previous section’s experiments, we use a linear decay function in WSD. All sweeps use a 0.6 0.6 B model with the Qwen3-0.6B architecture and global batch size 512. At 20 tokens per parameter (TPP), or 11.3 11.3 B tokens, we vary the effective peak from 0.008 0.008 to 0.024 0.024 in increments of 0.004 0.004 and the WSD decay ratio from 0.2 0.2 to 1.0 1.0 in increments of 0.2 0.2 . At peaks 0.008 0.008 and 0.012 0.012 , we additionally test TPP 50 and 100.

##### Observation 1: a larger peak LR calls for a longer decay.

In the left panel of Figure 6 , the best grid points at effective peaks 0.008 0.008 , 0.012 0.012 , 0.016 0.016 , 0.020 0.020 , and 0.024 0.024 are decay ratios 0.4 0.4 , 1.0 1.0 , 0.8 0.8 , 0.8 0.8 , and 1.0 1.0 . These point optima fluctuate because the long-decay region is broad. A more stable summary is the lower edge of the region within 0.01 0.01 validation loss of the minimum: it moves nondecreasingly as 0.4 0.4 , 0.4 0.4 , 0.6 0.6 , 0.6 0.6 , and 0.8 0.8 , as shown in Figure 18 . At peaks 0.020 0.020 and 0.024 0.024 , ratios 0.8 0.8 and 1.0 1.0 differ by only 0.0005 0.0005 and 0.0044 0.0044 loss. Thus an almost fully linear decay is already competitive near the high end of the tested peak range. The two-anchor interpolation in Figure 17 provides a complementary check: using only ratios 0.2 0.2 and 1.0 1.0 at peak 0.024 0.024 , it places the estimated minimum at ratio 0.85 0.85 within the same long-decay region.

##### Observation 2: a longer training horizon also calls for a longer decay.

At effective peak 0.012 0.012 , the center panel of Figure 6 shows the competitive region shifting toward longer decay. Its lower edge moves from 0.4 0.4 at TPP 20 to 0.6 0.6 at TPP 50 and 100, while the point optima are 1.0 1.0 , 0.8 0.8 , and 0.8 0.8 . At peak 0.008 0.008 , the lower edge remains 0.4 0.4 , although the point optimum moves from 0.4 0.4 to 0.6 0.6 . The latter TPP100 grid lacks ratio 0.2 0.2 , so we treat it only as supporting evidence. The complete peak- 0.012 0.012 lower-edge trace appears in Figure 18 ; the conclusion is a broad shift toward longer decay, not a precise monotone point optimum.

##### The production schedule combines these three practical features.

The right panel of Figure 6 shows the complete production effective-LR schedule. Here, we present the Hyperball weight LR, which is numerically the effective LR for wrapped matrices, instead of the base LR. Phase 1 uses the open-ended power schedule in Equation 10 , decreasing from approximately 5 × 10 − 2 5\times 10^{-2} to 1.04 × 10 − 2 1.04\times 10^{-2} over 438.8B. Its tail preserves a continuation path as the data distribution evolves [ Shen et al., 2024b ] . Phase 2 joins near 10 − 2 10^{-2} and uses a long linear decay over 960.0B rather than spending most of training in a stable segment followed by a short terminal drop. The design therefore combines continual trainability, a high initial effective LR, and a long final decay.

##### Multi-power law provides a back-of-the-envelope, cross-schedule estimator under limited compute.

Before a long run, exhaustively sweeping every decay ratio is impractical. Therefore, we fit a multi-power law (MPL) Luo et al. [2025b] as a diagnostic, using only the two endpoint schedules. The runs are at TPP 20 and with decay ratios 0.2 0.2 and 1.0 1.0 . Two curves per effective peak recover the increasing peak-to-decay trend: the estimated ratio rises from 0.33 0.33 at peak 0.008 0.008 to 0.85 0.85 at peak 0.024 0.024 . Although it is not a certificate of optimality, we use this result as a back-of-the-envelope trend diagnostic. The fitting protocol, unseen schedules, and errors are reported in Appendix F .

### 3.4 Curriculum Model Averaging

When open-source datasets provide usable sample-level scores, we rank samples within each source and construct a mixture-preserving curriculum that places more preferred samples later in training. This ordering is designed to improve the data efficiency of higher-quality samples. However, a conventional terminal learning-rate decay can work against this objective: examples presented near the end of the curriculum are also processed when parameter updates have become small [ Luo et al., 2026 ] . Curriculum Model Averaging (CMA) [ Luo et al., 2026 ] addresses this conflict by combining an ascending data curriculum with constant-LR training and averaging late checkpoints. Our production recipe adopts this principle in Phase 2. We first follow the scheduled learning-rate trajectory, then resume from a selected late checkpoint, hold the base learning rate fixed, and average six checkpoints from the continuation. The complete data and optimization flow is summarized in Figure 7 , with implementation details provided in Sections H.1 and H.3 . We next describe the curriculum construction, the selected constant-LR continuation with checkpoint averaging, and the corresponding ablation results.

For convenience, we label the Phase 2 variants: UD ( U niform data with LR D ecay) globally reshuffles the Phase 2 pool and follows the scheduled learning-rate decay. CD ( C urriculum data with LR D ecay) keeps that decay but presents the data pool in the component-local curriculum order. CDC ( C urriculum data, with LR D ecay then C onstant LR) uses the curriculum order, follows the decay, then continues at a constant learning rate, and reports the final checkpoint. CMA ( C urriculum M odel A veraging) is the production variant: it averages the last several checkpoints from the CDC recipe.

##### The data curriculum arranges the order within each component while keeping the target mixture weight.

We first establish a reproducible order separately for every retained component. If a component provides a usable sample-level quality signal, we sort its examples according to the source-specific interpretation of its score, placing less preferred score ranges earlier and more preferred ranges later. Components without a usable score instead follow a fixed random order. We then assign each example a normalized within-component rank according to its position in this ordered sequence, measured by cumulative token mass. Intuitively, this rank indicates which source-local score range an example falls into. For example, a rank of 0.25 0.25 means that approximately one quarter of the component’s tokens appear earlier in its ordered sequence.

We divide the normalized rank range into 376 intervals, with each curriculum bucket formed by taking the corresponding interval from every component. Each bucket contains approximately 2.5 2.5 B tokens, drawing roughly 1 / 376 1/376 of tokens from every component and preserving the selected cross-component mixture. As training advances through the buckets, scored components move toward their more preferred score ranges, while unscored components advance through their fixed random orders. Since scores are interpreted only within their own sources and are not compared across components, this construction is not a global quality ranking. The scalable approximation used to construct the 376 buckets and the exact transition schedule are given in Sections H.1 and H.2 and Table 19 .

##### The canonical CMA run uses the best-performing late-stage continuation with model averaging.

We compare three late-stage training strategies: averaging checkpoints directly from the terminal-decay trajectory, and two constant-LR continuations resumed from steps 215,000 and 218,000. Among these configurations, the continuation from step 218,000 gives the strongest observed performance and is therefore selected for the production model. For this continuation, training resumes from step 218,000 with the base learning rate fixed at 4.08 × 10 − 5 4.08\times 10^{-5} . With the production Hyperball multiplier m = 10 m=10 , the Hyperball weight LR is therefore 4.08 × 10 − 4 4.08\times 10^{-4} , which is also the effective LR for the MuonH-wrapped matrices. During the late stage of this continuation, we save checkpoints at approximately regular intervals and form the released model by equally averaging the parameters of six such checkpoints. Neighboring checkpoints are typically separated by 100 optimizer steps, or approximately 0.63 0.63 B tokens under the production batch configuration. Full resume, checkpoint-spacing, and averaging details are provided in Section H.3 .

##### Ablation study on curriculum, model average, and const-LR continuation.

The endpoint comparisons in Figure 13(b) support a structured comparison of the jointly selected recipe ingredients, as illustrated in Appendices G and 13(b) . First, curriculum ordering improves over uniform ordering by 1.18 1.18 points without model averaging (CD: 57.17 versus UD: 55.99) and by 1.61 1.61 points when comparing the model averages of the corresponding decay trajectories (CD model average: 57.18 versus UD model average: 55.57). Second, model averaging is not independently beneficial in both ordering conditions : it changes the UD endpoint by − 0.42 -0.42 points and the CD endpoint by + 0.01 +0.01 points without constant-LR continuation. Finally, the two CDC controls score 55.64 and 57.12 for the 215k and 218k continuations, respectively, when evaluated at their unaveraged final checkpoints. With six-checkpoint averaging, the corresponding branches reach 56.80 and 57.81. We therefore select the 218k averaged branch as the production CMA endpoint. It combines curriculum ordering, late constant-LR continuation and six-checkpoint averaging. Part of the observed advantage of CD over UD ordering could be attributed to benchmark contamination in the later-stage data, as discussed in Appendix A . But the remaining advantage is observed after post-training in Section 4.4 , and the contamination alone is unlikely to explain the persistent improvement. We report the exact base LR and Hyperball effective LR, checkpoint IDs, and the model-export procedure in Section H.3 .

### 3.5 Post-Training Recipe

##### Motivation and question.

In the above, we present a complete, low-cost pretraining recipe. The resulting pretrained checkpoints are later used for post-training. Here we focus on supervised fine-tuning (SFT). This raises a natural but underexplored question: does the choice of pretraining recipe affect model performance after SFT? A released checkpoint alone is usually not enough to answer this question. Even when a checkpoint is publicly available, its full training recipe and data may be unavailable. This makes it difficult and expensive to rerun the training and ablate specific pretraining choices. Small-scale ablations are easier, but the undertrained base models may be too weak to show a stable post-training performance [ Qi et al., 2025 ] . Therefore, our low-cost, fully open recipe allows us to study this question at the scale of the final model.

In particular, we compare the SFT results of checkpoints from two Phase 2 recipes: uniform data ordering with learning rate decay (UD) and Curriculum Model Averaging (CMA). The UD checkpoint performs worse on the base-model benchmark suite than the CMA checkpoint. Although both recipes use the same model architecture and Phase 2 training corpus, they differ in data ordering and checkpoint construction. This comparison therefore requires two separate Phase 2 training runs. In the CMA recipe, the curriculum places more high-quality data near the end of training. This changes the data distribution over the course of Phase 2. The base-model improvement may therefore be temporary and tied mainly to the final stage of pretraining. We test whether this improvement remains after both checkpoints undergo the same SFT procedure.

##### Data construction and experimental setup.

To study this question, we apply the same SFT procedure to the UD and CMA checkpoints for each data setup. All other training conditions are matched within the same setup. We continue to use the MuonH optimizer, as in pretraining [ Wen et al., 2026 ] . The base learning rate follows a cosine schedule that decays from 1 × 10 − 5 1\times 10^{-5} to 1 × 10 − 7 1\times 10^{-7} . The MuonH-managed matrix parameters use a 10 × 10\times multiplier to obtain their Hyperball weight learning rate. We set the global batch size to 160.

To examine the results under different SFT data distributions, we construct three data setups. (1) GSM8K-based SFT . It combines the original GSM8K training split with GSM8K-related synthetic data from MetaMathQA and OpenMathInstruct-2. It also includes selected Tulu-3 components, which cover mathematics and instruction following. (2) Math&Code SFT with replay . It contains a larger collection of mathematics data from GSM8K, MetaMathQA, OpenMathInstruct-1, and OpenMathInstruct-2. It also includes code data from OpenCodeInstruct and Magicoder. We add a small, shuffled replay set from the final part of the Phase 2 curriculum, which contains the highest-quality Phase 2 data. (3) Tulu-3 mixed-domain SFT . It uses the English portion of Tulu-3-sft-mixture-0225 [ Lambert et al., 2024 ] . Because this mixture covers several domains, it provides a broader test of general capability and instruction following.

For all three setups, we deduplicate the training examples and discard empty sequences. For decontamination, we remove any training example that shares a 13-gram with an evaluation test example. Detailed data composition and training settings are provided in Appendix E .

##### Evaluation.

We use the same evaluation protocol for the UD and CMA models within each setup. For GSM8K-based SFT and Math&Code SFT with replay, we evaluate the resulting models on GSM8K with the lm-evaluation-harness framework. Because generated answers can differ in format, we use flexible extraction to identify the final answer. We report the average result over three repeated seeds. For Tulu-3 mixed-domain SFT, we use OpenCompass to evaluate the resulting models on 15 tasks for base models and IFEval. The evaluation protocol of the 15 tasks is adapted to the fine-tuned models. This suite covers knowledge, reasoning, code, and instruction following. We report both the overall average and the individual task results, since the effect may differ across tasks. Further evaluation details are provided in Appendix E .

### 3.6 Data Recipe

Domain Phase 1 Phase 2 Tokens Share Tokens Share English 321.31B 73.2% 557.52B 59.4% Math 31.56B 7.2% 171.29B 18.3% Chinese 51.34B 11.7% 88.48B 9.4% Code 34.62B 7.9% 108.11B 11.5% SFT / instruction – 0.0% 12.66B 1.3% Total 438.84B 100.0% 938.06B 100.0%

Our goal is to select a compact set of high-quality and diverse data sources while preserving broad domain coverage. Candidate datasets differ in both quality and capability profile, so source-level metadata alone is not sufficient for data selection. We therefore use proxy benchmarking to compare candidate data slices under a shared protocol. As illustrated in Figure 9 , each run starts from the same Qwen3-0.6B checkpoint and follows the same continuation-training schedule. After continuation training, we evaluate the resulting checkpoint on a fixed 15-benchmark suite. The score vector from this evaluation serves as the capability profile of the candidate slice. These score vectors can guide our data recipe in heuristic. The following three paragraphs describe how we construct fine-grained slices, select and allocate data sources, and balance different capabilities in the final mixture. Detailed settings are provided in Appendix I .

#### 3.6.1 How Do We Select a Good Data Recipe?

##### Fine-grained data slice selection.

We construct candidate data slices according to source size and the availability of sample-level quality scores. We apply three sampling rules. First, if a dataset contains more than 50B tokens and provides usable score labels, we sort its examples by score and sample four local slices around the 0th, 25th, 50th, and 75th percentile positions. Second, for datasets between 5B and 50B tokens, as well as larger datasets without usable score labels, we randomly sample one 4B-token slice. Third, we do not evaluate the remaining smaller datasets. For each large scored source, the four slices form a within-source quantile benchmark. Because these slices come from different positions in the same score ordering, they reveal quality differences that a single source-level result would hide. For example, the q00 slice of DCLM-Dedup ranks third in average proxy performance, while its q25 slice ranks eleventh. This result shows that different regions of the same source can provide substantially different training value.

##### Feature-guided data selection.

We use the measured capability profiles to guide both source selection and data allocation. MegaMath-Web-Pro performs strongly on Math and General, while MegaMath-Code provides the strongest Code signal among the measured candidates. We therefore assign both sources larger shares in Phase 2 to strengthen general and target capabilities. The quantile results also guide within-source filtering. Because the top-scoring region of DCLM-Dedup performs much better than its lower-ranked regions, we retain 34B tokens from this region in Phase 2. This selection corresponds to approximately the top 5% of DCLM-Dedup. FineWeb-Edu-CN ranks highest on Chinese capability, so we also increase its share to preserve Chinese performance. The proxy results provide useful evidence for increasing, reducing, or filtering different data sources. Here, we take heuristics from these capability profiles and still choose the concrete mixing ratios manually.

##### Capability balance across domains.

The proxy results show that different datasets have substantially different capability profiles. A source that performs well on one group of benchmarks may perform less well on others. For example, FineWeb-Edu-CN ranks first on Chinese capability but is weaker on Math, Code, and General ( Appendix I ). We therefore need to balance broad general knowledge with more specialized reasoning capabilities rather than maximize a single benchmark dimension. To make these trade-offs more explicit, we apply PCA to the capability vectors. As shown in Figure 20 , General aligns mainly with PC1, while Code points in the opposite direction. Math aligns mainly with PC2 and remains slightly positive on PC1. In this projection, Code shows a stronger trade-off with General than Math does ( Appendix I ). This observation informs the roles of the two pretraining phases. Phase 1 preserves broad coverage, with English accounting for 73.2% of 438.8B ( Figures 8 and 8 ). Phase 2 then adds more specialist data, but allocates a slightly larger share to mathematics than to code. In this way, the two-phase recipe strengthens the target reasoning capabilities while retaining broad general coverage.

#### 3.6.2 How Do We Preprocess Data and Reproduce the Shards?

##### Data preprocessing pipeline.

The pipeline turns the candidate collection into fixed training shards in four stages. Within-source deduplication removes repeated documents from web components such as DCLM-Dedup and FineWeb-Edu-EN. We do not deduplicate across sources because our analysis found little additional reduction in data volume. Proxy benchmarking evaluates each eligible candidate source or within-source slice under the shared protocol described above and produces a multidimensional capability profile. Heuristic filtering combines these profiles with domain coverage, language balance, and token-budget constraints to determine the retained portion and mixture weight of each source. Final materialization freezes the source revisions, retention rules, mixture weights, and random seeds. It then merges the retained components into fixed shards that are read sequentially during training, without online mixing or sampling. In addition, we also use PreSelect fastText [ SHUM et al., 2025 ] to evaluate the Nemotron-HQ [ Mahabadi et al., 2025 ] and Nemotron-HQ-Synthetic datasets. Apart from this step, we do not perform additional compute-intensive data scoring and primarily rely on existing quality scores.

##### Kaiyuan-Spark preprocessing framework.

We implement the recipe with Kaiyuan-Spark , a Spark-based data preprocessing framework [ Luo et al., 2025a ] . The framework is designed for efficient large-scale data processing. For compute-intensive operations such as MinHash deduplication, it uses native C++ kernels through the Chukonu integration. This implementation is inherited from Chukonu, whose original Spark benchmark reports an approximately 2.5 × 2.5\times speedup over the corresponding JVM implementation [ Yu et al., 2021 ] . The framework also supports the curriculum bucket materialization required by the Phase 2 design. Given a fixed component pool and mixture, it assigns normalized progress within each component, aligns the resulting slices into token-sized buckets, and materializes their order using recorded random seeds. The same component pool can therefore produce either an ordered curriculum stream or a uniformly ordered stream by changing only the materialization order. These two data streams support the UD, CD, and CMA recipes in Phase 2 ( Section 3.4 ).

##### Two-phase data and transition schedule.

All training data are materialized as static shards and manifests before training. Production runs therefore do not rely on online scoring or dynamic mixture sampling. The Phase 1-to-Phase 2 transition spans approximately 43.9 43.9 B training tokens. During this transition, the sampling weight of the Phase 1 replay mixture decreases linearly , while that of the Phase 2 mixture increases linearly . The transition therefore consumes approximately 21.9 21.9 B tokens from the Phase 1 replay data and 21.9 21.9 B tokens from the beginning of the Phase 2 data pool. The full Phase 2 component pool contains approximately 938.1 938.1 B tokens, including the 21.9 21.9 B tokens consumed during the transition. After the transition, the remaining 916.1 916.1 B Phase 2 tokens are consumed without Phase 1 replay. Counting the transition, the Phase 2 training stage consumes approximately 960.0 960.0 B tokens in total: 21.9 21.9 B tokens of Phase 1 replay and 938.1 938.1 B tokens from the Phase 2 pool. Together with Phase 1 training, the complete pretraining run consumes approximately 1.4 1.4 T tokens.

## 4 Evaluation

### 4.1 Evaluation Setup

#### 4.1.1 Model Selection

We compare Puro-2B with recent language models of comparable size, primarily in the 1B–3B parameter range. We group these baselines according to the scope of their public releases. Open-weight models make their weights publicly available but do not release the complete training data or recipe, whereas open-recipe models additionally release the data and training artifacts needed to study or reproduce training.

Open-weight models. • Qwen series [ Yang et al., 2024a , Yang et al., 2024b , Yang et al., 2025a , Qwen Team, 2026 ] : We include Qwen2-1.5B, Qwen2.5-1.5B, Qwen3-1.7B-Base, and Qwen3.5-2B-Base, which provide successive generations of strong compact foundation models in the 1.5B–2B range.

• Gemma series [ Rivière et al., 2024 , Gemma Team, 2025 , Gemma Team, 2026 ] : We select Gemma-2-2B, Gemma-3-1B-PT, and Gemma-4-E2B-Base to cover three generations of Google’s lightweight open models.

• Llama 3.2 [ Meta AI, 2024b ] : We include Llama-3.2-3B, the latest pretrained Llama checkpoint available in the compact 1B–3B range.

• LFM2.5 [ Liquid AI, 2025 ] : We include LFM2.5-1.2B-Base as a recent compact base model designed for efficient on-device deployment.

• Falcon-H1 [ Zuo et al., 2025 ] : We evaluate the standard and deeper 1.5B base variants as efficiency-oriented hybrid-model baselines.

Open-recipe models. • Instella [ Liu et al., 2025 ] : Instella-3B was trained entirely on openly available data, with model weights, training data, configurations, and code released to support reproducibility. Despite using fewer pretraining tokens than many contemporary models, it is reported to remain competitive with leading open-weight models at a similar scale.

• OLMoE [ Muennighoff et al., 2025 ] : OLMoE-1B-7B-0125 is a sparse model with 1B active and 7B total parameters. Its release includes intermediate checkpoints, training data, code, logs, and evaluation resources. The model is positioned as a strong performance–efficiency baseline among fully-open mixture-of-experts models.

• Yulan-Mini-2.4B [ Yiwen et al., 2025 ] : Yulan-Mini-2.4B is a 2.4B model trained on 1.1T tokens, with its training recipe and phase-wise data composition publicly documented. It emphasizes data-efficient pretraining and reports competitive performance against similarly sized models trained with substantially more data.

• SmolLM3 [ Bakouch et al., 2025 ] : SmolLM3-3B-Base was pretrained on approximately 11T tokens, with the public data mixture and training configurations released alongside the model. It is presented as a strong fully-open model at the 3B scale, with performance competitive with several larger compact models.

• MiniCPM5 [ OpenBMB Team, 2026 ] : MiniCPM5-1B-Base is a compact dense model developed for local and resource-constrained deployment. Its associated web and mathematical training corpora are publicly released, while the MiniCPM5 family targets strong reasoning, coding, and mathematical capabilities within a small deployment footprint.

For a fair comparison, we consistently evaluate the pretrained/base versions of all models.

#### 4.1.2 Benchmarks

Our evaluation covers 15 benchmarks, organized into two groups: mathematics and code, and reasoning and knowledge. For mathematical reasoning, we use GSM8K [ Cobbe et al., 2021 ] and the original MATH benchmark [ Hendrycks et al., 2021b ] , which measure grade-school arithmetic and competition-level mathematics, respectively. For code generation, we use sanitized-MBPP [ Austin et al., 2021 ] and HumanEval [ Chen et al., 2021 ] , where generated Python programs are verified by unit tests. For reasoning and knowledge, we use MMLU [ Hendrycks et al., 2021a ] , MMLU-Pro [ Wang et al., 2024b ] , ARC-Challenge and ARC-Easy [ Clark et al., 2018 ] , BoolQ [ Clark et al., 2019 ] , CommonsenseQA (CSQA) [ Talmor et al., 2019 ] , HellaSwag [ Zellers et al., 2019 ] , PIQA [ Bisk et al., 2020 ] , SocialIQA (SIQA) [ Sap et al., 2019 ] , WinoGrande [ Sakaguchi et al., 2020 ] , and Big-Bench Hard (BBH) [ Suzgun et al., 2023 ] . Together, these benchmarks assess mathematical reasoning, code generation, academic knowledge, reading comprehension, commonsense understanding, and general multi-step reasoning.

#### 4.1.3 Implementation Details

We evaluate every model with OpenCompass [ Contributors, 2023 ] using a fixed checkpoint and tokenizer revision, dataset snapshot, benchmark-specific prompt, shot examples, generation configuration, and answer postprocessor. GSM8K, MATH, sanitized-MBPP, HumanEval, MMLU-Pro, and BBH use generation-based (GEN) evaluation: the model produces a free-form response from which the final answer or executable code is parsed. GEN decoding is greedy with fixed maximum output and stopping rules. The remaining nine benchmarks use perplexity-based (PPL) evaluation, which ranks the provided candidate answers by their fixed token log-likelihoods and therefore introduces no generation randomness. For GSM8K, a shared deterministic postprocessor prioritizes explicit final-answer markers and boxed answers, preserves signs and fractions, and uses the final numeric expression only when no explicit answer is present.

Following the OLMES [ Gu et al., 2025 ] convention, for tasks supporting both cloze formulation (CF) and multiple-choice formulation (MCF), we evaluate both formulations for each model and use the better-performing formulation when computing the reported aggregate. This selection is performed independently for each model. All scores are percentages. In each table, Avg is the unweighted arithmetic mean over the benchmarks displayed in that table. Throughout the cost–performance and scaling analyses, overall performance P P denotes the unweighted arithmetic mean over all 15 benchmark scores reported in Tables 3 and 4 . Every score in our tables is produced by this common, deterministic pipeline.

### 4.2 Resulting Performance

##### Mathematics and code.

As shown in Table 3 , Puro-2B obtains an average score of 43.50 across the four generative benchmarks. It exceeds Qwen2-1.5B by 3.21 points and is within 4.02 points of Qwen2.5-1.5B. Among the open-recipe baselines, Puro-2B outperforms Instella-3B, OLMoE-A1B/7B, and MiniCPM5-1B-Base, while Yulan-Mini-2.4B, SmolLM3-3B-Base, and the newly included MobileLLM-R1-950M-base achieve higher averages. MobileLLM-R1 is a strong math/code-oriented compact baseline in this panel, exceeding Puro-2B by 3.86 points on the four-task average. Given these differences in model size and capability emphasis, Puro-2B remains a competitive open-recipe model for mathematics and code generation at a comparatively small scale.

##### Reasoning and knowledge.

Table 4 shows that Puro-2B achieves a mean score of 63.02 over the eleven reasoning and knowledge benchmarks. It outperforms Qwen2-1.5B by 2.48 points and is within 2.51 points of Qwen2.5-1.5B. Within the open-recipe group, Puro-2B surpasses OLMoE-A1B/7B, Yulan-Mini-2.4B, MiniCPM5-1B-Base, and MobileLLM-R1-950M-base, while trailing the larger Instella-3B by only 0.11 points. SmolLM3-3B-Base achieves a higher average, but it is also a larger model. These aggregate results show that Puro-2B remains competitive with larger open-recipe models and retains a 6.29-point reasoning/knowledge advantage over MobileLLM-R1. Together with the reduced training cost enabled by low-precision training, this performance indicates a favorable balance between broad capability and replication cost.

### 4.3 Cost Estimation

#### 4.3.1 Cost-Saving Factors

We estimate four cost factors separately: hardware price efficiency, FP8 throughput, MuonH quality-equivalent compute, and the joint Phase 2 recipe. Each factor uses its own reference comparison, so the reported values are not measurements of one sequential end-to-end speedup. The product of these components can only be treated as illustrative transfers. Hardware uses a specification-and-price accounting proxy, blockwise FP8 uses quality-adjusted throughput with MuonH fixed, and the complete MuonH recipe uses quality-equivalent compute with blockwise FP8 fixed. Phase 2 first fits a scaling law of compute cost using the UD recipe ( Figure 2(b) ), then measures the equivalent costs of the UD recipe for the CD and canonical CMA endpoints to quantify the corresponding gains. These comparisons do not form a joint factorial ablation due to different references, so their products are accounting estimates rather than directly measured end-to-end speedups 9 9 9 For example, MuonH ablation and FP8 ablation use token-per-parameter of 20 (TPP=20) while the production training is an overtrained setup (TPP=700). We run the alternative compute-optimal setup due to limited compute. .

For the ladder evidence below, C = 6 ​ N ​ D C=6ND denotes theoretical training compute, where N N is the scaling parameter count and D D is the number of training tokens. A compute-equivalent multiplier κ \kappa means that a variant trained with C / κ C/\kappa reaches the fitted loss of its baseline at C C . For precision, we define BF16-equivalent compute retention as ρ C = C BF16 / C FP8 \rho_{C}=C_{\mathrm{BF16}}/C_{\mathrm{FP8}} at matched fitted loss; this converts the vertical precision gap into a horizontal compute adjustment.

##### RTX 5090 exhibits a high cost-performance ratio and utilization.

Under the specifications and prices in , the RTX 5090 proxy provides 2.77 × 2.77\times the peak BF16 compute per unit price of H200 and 2.74 × 2.74\times the peak FP8 compute per unit price of H200. Moreover, with training configuration tuning, as shown in Section 3.1.3 , our recipe can achieve 73% MFU in mixed-precision training, making good use of the RTX 5090’s capability.

##### MuonH provides a quality-equivalent compute shift in the matched ladder.

Under the definition above, the quality-equivalent compute reduction is 1 − 1 / κ 1-1/\kappa . The matched scaling ladder in Figure 10(a) estimates a 1.19 × 1.19\times compute-equivalent multiplier for the complete MuonH recipe relative to Muon, corresponding to 16.1% less theoretical compute at matched validation loss. The scaling estimation is performed over TPP=20 ladder with a shared-shape fit to transfer to production run. At the production run of 2B parameters and 1.4T tokens, using this efficiency estimation, the MuonH optimizer can use 1.41 × 10 22 1.41\times 10^{22} FLOPs to match 1.68 × 10 22 1.68\times 10^{22} FLOPs with the Muon baseline.

##### Blockwise FP8 retains a stable quality fraction while accelerating the 2B proxy.

A precision format may change both execution rate and model quality, so its net efficiency multiplier is the product of throughput speedup and effective-compute retention. As summarized in Figure 10(b) , blockwise FP8 has a small validation-loss gap at all five ladder scales. The shared-shape fit gives ρ C = 98.0 % \rho_{C}=98.0\% . Matching BF16 quality therefore requires approximately 2.0% additional nominal compute. At 1.7B, the closest throughput proxy to the 2B production model, median throughput improves by 1.36 × 1.36\times ; charging the quality penalty yields a 1.34 × 1.34\times net speedup, or 25.2% fewer GPU-hours at matched quality. Then, by this estimation, the 2B-parameter, 1.4T-token canonical run would consequently use 22,654 FP8 GPU-hours to achieve an equivalent performance using approximately 30,286 BF16 GPU-hours.

##### The Puro Cost Scaling Law summarizes recipe-specific scale-down behavior.

Starting from the same Phase 1 checkpoint, we continue Phase 2 training with UD at several budgets and use these runs to characterize the empirical cost–performance relationship, which we refer to as the Puro Cost Scaling Law. It is a scale-down relationship for this recipe, not a universal law across model families. Here, we focus on the scale-down trend to help the community with limited compute and time budgets decide a feasible performance target. We fit P = a + b ​ log 2 ⁡ ( C − C P1 ) P=a+b\log_{2}(C-C_{\mathrm{P1}}) , where P P is the unweighted arithmetic mean over the 15 benchmarks reported in Tables 3 and 4 , C C is total reproduction cost, and a a and b b are the fitted intercept and slope. The checkpoint-level values of P P used in the fit are reported in Table 18 . Costs shown in the figure use the USD conversion in Table 5 . We set C P1 = USD ​ 1.84 C_{\mathrm{P1}}=\text{USD }1.84 K from the measured Phase 1 active-training GPU-hours and hold it fixed while fitting performance against the incremental Phase 2 cost. The observed UD checkpoint at approximately $4.4K already exceeds Qwen2-1.5B without CMA. This shifted fit reduces in-sample RMSE from 0.452 for the unshifted log-cost fit to 0.209. Inverting it at the CD endpoint score places its UD cost equivalent at approximately $11.36K, or 1.65 × \times the production recipe’s $6.89K rental-equivalent estimate from measured active-training GPU-hours. Inverting the same fit at the reported Puro-2B aggregate, which additionally includes a constant-LR continuation and averages the last six checkpoints, gives approximately $16.55K, or 2.40 × \times the production cost. The canonical CMA endpoint optimizes ordering, continuation, and checkpoint averaging together. We therefore report its fitted cost equivalent as a recipe-level point estimate rather than as a curriculum-only gain or a universal cost law.

#### 4.3.2 Cross-Model Reproduction Cost

##### Estimation protocol.

For Figure 1 , we prioritize reported monetary cost or accelerator usage. We use reported GPU-hours, or GPU-hours precisely derived from other reported statistics, and convert them with the reference rental rates listed in Table 5 in the appendix. When no such GPU-hour basis is available, we use the token-based route and estimate compute as C = 6 ​ N ​ D C=6ND , using the reported accelerator and MFU when available and H100-equivalent GPU-hours at 70% MFU otherwise. Models that disclose neither source are omitted. For Puro-2B , we use measured active-training GPU-hours: 6,009.46 GPU-hours for Phase 1 and 16,504.95 GPU-hours for Phase 2. We convert their sum using the RTX 5090 rate in Table 5 . Across all models, this hierarchy distinguishes reproduction-cost estimates based on reported monetary costs or accelerator usage ( reported compute ) from token-based estimates ( estimated compute ). The complete coordinate construction, including the treatment of special cases, and the Pareto-frontier procedure are given in Appendix B .

##### Pareto-frontier result.

Under the stated accelerator-cost assumptions, Puro-2B appears above and to the left of the comparison-model frontier in Figure 1 , combining higher average performance with a lower estimated replication cost. This advantage is clearest relative to the open-recipe models.

### 4.4 Post-Training Results and Analysis

We conduct an ablation study of the two Phase 2 designs to examine how the pretraining recipe affects the results after SFT. The uniform data ordering with learning-rate decay (UD) recipe trains the model on globally reshuffled Phase 2 tokens, while the curriculum model averaging (CMA) recipe follows a data curriculum, keeps the learning rate constant near the end of training, and averages the final checkpoints ( Section 3.4 ). We apply the same SFT procedure to the two pretraining checkpoints and compare the resulting UD-based and CMA-based models. The experimental setup is detailed in Section 3.5 and we analyze the results in this section.

##### GSM8K-based SFT.

The GSM8K-based SFT experiment compares the UD and CMA checkpoints without pretraining replay. After SFT, the UD-based model reaches 66.89% mean GSM8K accuracy, while the CMA-based model reaches 68.66%. On average over three random seeds, the CMA-based model is ahead by 1.77 percentage points ( Figure 11(a) and Table 12 ). We use flexible answer extraction to limit score differences caused by output formatting. We then compare the sets of questions answered correctly by the two models. Under this protocol, the CMA-based model still solves more questions that the UD-based model misses. The remaining advantage is therefore better explained by stronger mathematical capability than by answer formatting.

##### Math&Code SFT with replay.

The Math&Code SFT with replay experiment extends the comparison to a larger mathematics mixture, together with code data and a small replay component. It also uses a longer SFT process. One possible explanation for the base-model difference is a temporary advantage from placing higher-quality data near the end of pretraining. A longer SFT process on a different data distribution tests whether this advantage persists. After this longer SFT process, the UD-based model reaches 74.10% mean GSM8K accuracy, while the CMA-based model reaches 76.12%. On average over three random seeds, the CMA-based model is ahead by 2.02 percentage points ( Tables 3 and 13 ). The CMA advantage therefore remains after more SFT updates on a broader data distribution. The gap is also slightly larger than in the GSM8K-based SFT setting.

##### Tulu-3 mixed-domain SFT.

The first two setups focus primarily on mathematics. To examine whether the difference extends beyond this domain, we use Tulu-3 mixed-domain SFT and evaluate the resulting models on 15 benchmarks ( Tables 3 and 4 ). As shown in Figure 11(b) , the CMA-based model scores 1.17 points higher on average and performs better on 10 of the 15 benchmarks. The aggregate advantage therefore extends beyond mathematics, although the gains are not uniform across tasks. The task-level results show a different balance across capabilities rather than a uniform improvement. We separately evaluate instruction following on IFEval, where the CMA-based model scores 1.36 points higher than the UD-based model. The results are averaged over three repeated runs, which show the same overall trend. Further details are provided in Appendix E .

Taken together, the three experiments show that the difference between the two pretraining recipes remains after increasingly broad SFT. The CMA-based models retain stronger mathematics performance in both the GSM8K-based SFT and Math&Code SFT with replay settings. They also maintain a higher aggregate score after Tulu-3 mixed-domain SFT and perform better on instruction following. Because the task-level gains are uneven, these results do not imply that CMA improves every capability uniformly. This variation makes more controlled curriculum design an important direction for future work.

## 5 Related Works

### 5.1 Open-Recipe Language Models

The core science and engineering of large-scale pretraining remain difficult to study when leading systems are either closed or release weights without the data and training recipe needed to reconstruct the run [ Luo et al., 2025a ] . The term “open model” therefore covers substantially different release practices. Widely used model families such as Qwen [ Yang et al., 2024a , Yang et al., 2024b , Yang et al., 2025a , Qwen Team, 2026 ] , Gemma [ Rivière et al., 2024 , Gemma Team, 2025 , Gemma Team, 2026 ] , Llama [ Meta AI, 2024b ] , LFM [ Liquid AI, 2025 ] , and Falcon [ Zuo et al., 2025 ] publish model weights and enough implementation information for evaluation and deployment. However, their exact pretraining corpora, sample order, and complete training state are not released, so an independent group cannot reconstruct the original training run. We refer to these releases as open-weight . In contrast, a open-recipe release aims to expose the model weights, training data or a reconstructible data recipe, training and evaluation code, and sufficiently detailed configurations and intermediate artifacts to support scientific inspection and reproduction [ Groeneveld et al., 2024 , Walsh et al., 2025 ] . Such releases turn a final checkpoint into a reproducible account of how the model was built, making controlled study possible for resource-limited research communities.

Open-recipe projects provide the broader experimental stack needed to study pretraining itself. Pythia releases ordered data and dense intermediate checkpoints; OLMo and OLMoE release data, training and evaluation code, checkpoints, and logs; and projects such as SmolLM3, Yulan-Mini-2.4B, and Instella provide public data mixtures and reproducible training configurations [ Biderman et al., 2023 , Groeneveld et al., 2024 , Muennighoff et al., 2025 , Bakouch et al., 2025 , Yiwen et al., 2025 , Liu et al., 2025 ] . These resources allow other researchers to investigate learning dynamics, data selection, scaling, and optimization without relying on undocumented industrial pipelines. Open-recipe models can also be competitive: as shown in Tables 3 and 4 , models such as SmolLM3-3B, Yulan-Mini-2.4B, and Instella-3B match or surpass several open-weight models of comparable scale on parts of our evaluation, making this direction important beyond reproducibility alone.

This openness and capability have nevertheless often required substantial resources. Yulan-Mini-2.4B was trained on 48 A800 GPUs, SmolLM3 on 384 H100 GPUs, and the first stage of Instella-3B on 128 MI300X GPUs [ Yiwen et al., 2025 , Bakouch et al., 2025 , Liu et al., 2025 ] . Kaiyuan-2B, our previous work, provides a more resource-conscious example by improving data and training efficiency on Ascend 910A hardware [ Luo et al., 2025a ] . These examples expose a remaining gap between openness and practical accessibility: a training pipeline may be fully reproducible yet remain too costly for small research groups to rerun. Addressing this gap requires considering training workload, numerical precision, hardware accessibility, and explicit cost accounting together.

### 5.2 Low-Cost Language Model Pretraining

Training cost is meaningful only under an explicit accounting boundary. Token count and FLOPs alone capture only part of pretraining cost. End-to-end replication cost also depends on the workload required to reach a target quality, realized hardware throughput, and hardware price. We therefore consider cost jointly with model quality rather than using any single measure of compute or hardware efficiency as a proxy for low-cost training.

Low-precision training primarily reduces the cost of executing a fixed workload. FP8 mixed precision accelerates the matrix multiplications that dominate Transformer training while retaining higher precision where needed. DeepSeek-V3 shows that this design can remain stable at large scale [ Micikevicius et al., 2022 , DeepSeek-AI et al., 2024 ] . Quartet II extends this direction to NVFP4 and improves its numerical behavior [ Panferov et al., 2026 ] . These methods improve execution efficiency for a given training workload, but do not by themselves reduce the token or nominal FLOP budget required to reach a target quality.

A complementary line of work reduces the training workload through model architecture, data selection, staged training, and systems optimization. TinyLlama, JetMoE, Yulan-Mini-2.4B, and Kaiyuan-2B explore different combinations of these techniques [ Zhang et al., 2024a , Shen et al., 2024a , Yiwen et al., 2025 , Luo et al., 2025a ] . Collectively, these projects show that useful small-model capability can be obtained under substantially smaller compute budgets than those of large-scale foundation models. The reported runs, however, generally rely on specialized data-center accelerators, so the hardware barrier for a small research group remains.

Hardware accessibility introduces a separate constraint. QLoRA reduces the memory required to fine-tune a pretrained model, leaving the original pretraining cost unchanged [ Dettmers et al., 2023 ] . LLMQ studies full-model pretraining on RTX 4090 GPUs using FP8, recomputation, sharding, and CPU offloading [ Schultheis and Alistarh, 2025 ] , while Quartet II measures the throughput benefit of NVFP4 on a single RTX 5090 [ Panferov et al., 2026 ] . The studies considered here establish the feasibility of relevant training operations on consumer GPUs, but focus primarily on systems feasibility or throughput rather than on releasing a trillion-token-scale base model with an explicit replication-cost boundary.

Puro-2B lies at the intersection of these directions. It is a from-scratch 2B base model trained on more than 1.4T tokens using FP8 on a multi-node RTX 5090 cluster, with measured accelerator usage and an explicit replication-cost boundary. We therefore evaluate the training recipe and hardware platform end to end, measuring both the resulting model quality and replication cost. Together, these components provide researchers with a reproducible and cost-accessible end-to-end pipeline for studying pretraining behavior and testing alternative methods at meaningful scales.

## 6 Conclusion and Future Direction

In this report, we present a cost-efficient, hardware-accessible reproducibility recipe for language model pretraining, and use it to train the Puro-2B model collection from scratch on consumer-grade RTX 5090 GPUs. Across different training budgets and recipe variants, the collection provides a direct view of the cost–performance tradeoff of our pipeline. A model from this collection reaches the performance of Qwen2-1.5B at about $4.4K, while our best checkpoint approaches Qwen2.5-1.5B at the canonical accelerator-only cost of about $6.9K. Based on these runs, we further formulate the Puro Cost Scaling Law to characterize how model capability scales down with training cost under our recipe. More broadly, these results demonstrate that useful from-scratch pretraining at the billion-parameter scale is feasible with modest budgets and accessible hardware.

There remain several directions for extending this foundation. First, we plan to extend the reproducibility recipe beyond pretraining to post-training, with particular emphasis on developing and studying agentic capabilities. Second, we plan to broaden the architectural space beyond standard dense Transformers, including looped Transformers, linear-attention models, mixture-of-experts architectures, and other emerging designs. Third, we aim to expand the hardware recipes beyond RTX 5090 GPUs and cover a wider range of training budgets, making it easier to study how the optimal system and training choices change across hardware and scale. We view Puro-2B not as an endpoint, but as evidence that affordable and inspectable model training is practical today, and as a baseline for future efforts to push the frontier toward lower cost, broader accessibility, and higher efficiency.

## 7 Acknowledgments

The authors would like to thank Yanfu Investments for providing computational resources. Zhenbo Sun and Chenyi Dang contributed during the early stages of this work. The authors also thank Kaiyue Wen, Kexian Tang, Minxing Yang, Zhan Ling, Haodong Wen, Shaowen Wang, Huaqing Zhang and Shuo Huang for their valuable feedback and insightful discussions, and Yanzheng Cai, Yuanwei Wang, Zhixuan Pan, Rui Chen, and Nuo Chen for their assistance with proofreading and manuscript revision. Multiple LLM-based tools were used to assist with drafting, proofreading, and language refinement.