Puro-2B: Poor Lab's Qwen2-1.5B Trained on RTX 5090 within $5090 — part 2/7

Cleaned-text code points [15588, 31304). Bibliography omitted here; source.md includes it.

Across both phases, we use Hyperball optimization for selected approximately scale-invariant matrices. MuonH updates these matrices and projects each one back to its initial Frobenius radius after every step, while AdamW updates the remaining parameters ( Section 3.3 ). Both parameter groups share a base LR schedule. MuonH group uses a 10 times base LR schedule as Hyperball weight LR. The base schedule follows a power schedule [ Shen et al., 2024b ] in Phase 1 and continues from its terminal Phase 1 value with linear decay along the main Phase 2 trajectory. The late CMA continuation instead holds the base LR fixed at its resume-point value before checkpoint averaging ( Section 3.4 ). Both phases use a sequence length of 4,096 tokens and a global batch size of 1,536 sequences. The optimizer configuration, learning rate multipliers, curriculum construction, transition schedule, checkpoint-averaging rule, and exact token accounting are reported in Sections 3.3 and 3.4 , Appendix C , and Table 7 .

##### Data acquisition, filtering, and mixture ( Section 3.6 ).

We construct the training corpus from publicly accessible datasets with licenses or terms that permit their use, enabling the corpus to be reproducibly reconstructed from the original sources. The corpus spans web text, code, and mathematics, and also includes synthetic examples. We first use our Kaiyuan-Spark framework ( Section 3.6.2 ) to deduplicate the large-scale web portion. Because the available data far exceeds our training budget, we must select source datasets, filter samples within each source, and combine the retained portions to improve overall data utility and match the desired capability profile. However, it is difficult to design a single sample-level quality score that is comparable across domains, apply it to every sample, and use it to determine which samples to retain. Instead, we run proxy experiments on candidate sources and representative data slices ( Section 3.6.1 ). Specifically, we train controlled small-scale models on these sources or slices and evaluate the resulting models on downstream tasks. The resulting vector of benchmark scores defines the proxy profile of each source or slice, which guides source selection and mixture design ( Section 3.6.1 ). For example, when stronger mathematical capability is desired, we can assign higher weights to datasets whose proxy profiles show stronger performance on mathematics benchmarks. Some open-source datasets also provide sample-level quality scores. Although these scores are not directly comparable across sources, they can support within-source filtering. For each such dataset, we sample proxy slices from different score ranges, compare their proxy profiles, and use the results to select a dataset-specific threshold for retaining high-scoring samples. The final domain-level recipe for the two pretraining phases is summarized in Figure 8 and Figure 8 , while reports family-level retained token counts, selection modes, assigned phases, source identifiers, and license terms rather than a complete component-level manifest.

##### Evaluation and post-training ( Sections 3.5 and 4 ).

Our evaluation combines a broad pretrained-model comparison with controlled tests of downstream adaptation. We first evaluate Puro-2B against open-weight and reproducible open-recipe models of comparable scale on general reasoning, mathematics, and coding. Under the report’s common budget accounting, Puro-2B surpasses Qwen2-1.5B and Gemma-2-2B while using less than one sixth of the stated training cost of comparable open recipes. We then use SFT as a probe of how the Phase 2 recipes transfer through post-training. Within each paired experiment, the CMA and UD initializations receive identical supervised data and optimization; mathematics checkpoints use one frozen generation and extraction protocol, while broad tasks use benchmark-appropriate prompts and scorers. The three settings are GSM8K-based SFT, Math&Code SFT with replay, and Tulu-3 mixed-domain SFT, with complete definitions and results in Section 3.5 and Appendix E .

### 2.2 Reproduction-Cost Boundary

Throughout this report, reproduction cost refers specifically to the compute cost of rerunning the finalized two-phase pretraining recipe once. The accounting boundary begins after the phase-specific training shards have been materialized and includes only the compute time of the Phase 1 and Phase 2 production runs. We report measured GPU-hours separately and convert them using the normalized RTX 5090 rental-equivalent rate specified in Section 4.3 and Appendix B . The ledger retains CNY(RMB) for traceability, while the main cost figures use the corresponding USD values. Both are reproducible cost proxies under the stated rate, not the authors’ total cash expenditure.

We do not estimate costs outside these production pretraining runs. The headline therefore excludes data acquisition and preprocessing, proxy-model experiments, scaling and ablation studies, failed or exploratory runs, post-training, evaluation, checkpoint averaging, and research labor. It also excludes non-accelerator resources such as CPU processing, storage, and networking, as well as taxes, depreciation, and other ownership costs not represented by the rental-equivalent rate. These exclusions do not imply that the activities are free; they mean that the reported number should be interpreted as a narrowly defined marginal accelerator cost for reproducing the final pretraining run, rather than as the total cost of developing the model or reconstructing every artifact from scratch.

## 3 Training Recipe

We organize the training recipe as follows. We first describe the infrastructure design that supports our training setup. We then present the pretraining recipe, focusing on optimization and data curriculum. Next, we explain how the training dataset is constructed. Finally, we present a post-training case study to demonstrate how our recipe enables end-to-end approach comparison.

### 3.1 Training Infrastructure

#### 3.1.1 RTX 5090 as a Cost-Effective GPU Choice

To make our recipe cost-efficient and the hardware accessible to the broader community, we made a non-standard infrastructure choice: we run all pretraining and post-training on a cluster equipped with consumer-grade GPUs (RTX 5090), instead of renting conventional data-center-oriented accelerators. Although data-center GPUs are generally expected to be more stable and compute-efficient, our cost analysis showed that RTX 5090 is a better fit for this workload.

{narrowtalltblr} [ caption = Peak hardware capability and cost efficiency of candidate NVIDIA GPUs., label = tab:gpu-cost-efficiency, notea = All performance numbers are from official NVIDIA documentation / datasheets / blog posts Andersch et al. [2022] , Corporation [2024] , Corporation [2025a] , Corporation [2025b] and refer to dense matrix operations with FP32 accumulation., noteb = The prices do not include CPU / storage / network, as they are marginal and often free on GPU rental platforms. A100, H200, and RTX PRO 6000 prices were collected from a public GPU rental platform on 16 Aug 2026. RTX 5090 has no widely available public rental channel due to NVIDIA EULA restrictions; we estimate its effective price by amortizing our hardware and electricity costs over five years. An 8-GPU node costs approximately 12,000 CNY ($1763) per month for our setup. Some private compute providers confirmed similar rental pricing for RTX 5090., notec = Hopper FP8 Tensor Cores use a reduced-precision path for nominal FP32 accumulation, with 22 effective precision bits Zhang et al. [2025a] , which may affect numerical accuracy., noted = PCIe Gen5 × \times 16 provides approximately 64 GB/s of theoretical bandwidth per direction, or approximately 128 GB/s of aggregate bidirectional bandwidth. Driver tweaks needed to achieve this bandwidth are discussed in Section 3.1.2 ., ] colspec = l r r r c c c c c r r r, rowhead = 2, row1-2 = font= , bg=gray!10 , cell11 = r=2l, cell12 = c=3c, cell15 = r=2c, cell16 = r=2c, cell17 = r=2c, cell18 = r=2c, cell19 = r=2c, cell110 = c=3c, hline3 = 0.5pt, GPU Model Peak tensor performance (TFLOPS) \TblrNote a Memory capacity Memory bandwidth TDP Intra-node P2P BW (Bi-Di) Single-GPU price \TblrNote b Peak compute/$ (EFLOP/USD) BF16 FP8 NVFP4 BF16 FP8 FP4 A100 SXM4 312 N/A N/A 80 GB 2 TB/s 400 W 600 GB/s $1.79/h 0.63 N/A N/A H200 SXM5 989.5 1979 \TblrNote c N/A 141 GB 4.8 TB/s 700 W 900 GB/s $4.00/h 0.89 1.78 N/A RTX PRO 6000 503.8 1007.6 2015.2 96 GB 1.8 TB/s 600 W 128 GB/s \TblrNote d $1.89/h 0.96 1.92 3.85 RTX 5090 209.5 419 1676 32 GB 1.8 TB/s 575 W 128 GB/s \TblrNote d $0.31/h 2.43 4.87 19.46

shows the features and prices of different generations of NVIDIA GPUs. RTX 5090 has a clear gap in absolute peak throughput compared with data-center GPUs, especially because its Tensor Cores are artificially limited to half of actual peak performance when using FP32 accumulation. However, its much lower effective price gives substantially higher compute per dollar: its BF16 and FP8 cost efficiency is about 2.7 × \times that of H200. Moreover, because policy and cost constraints made Blackwell data-center accelerators such as GB200 unavailable to us, RTX 5090 is also the most practical option with FP4 support.

RTX 5090 still has significant hardware limitations, including smaller memory capacity and the lack of NVLink for high-bandwidth GPU-to-GPU scaling. These drawbacks cannot be ignored. Nevertheless, our target model is small enough that, with the hardware configuration and parallel training setup described below, the RTX 5090 cluster reaches a mixed-precision effective MFU of approximately 73% under the stated precision-weighted peak convention. Section 3.1.3 defines the convention and measurement boundary; this is not a BF16-only full-scale result. Its cost advantage therefore remains valid for our setting.

#### 3.1.2 Hardware Setup and Tweaks

Our training cluster consists of multiple x86 GPU servers. Each server is equipped with dual CPUs, eight RTX 5090 GPUs (four in each CPU socket), and sufficient host memory. The performance numbers in Section 3.1.1 are single-GPU metrics; for multi-GPU training, both intra-node P2P bandwidth and inter-node network bandwidth must be considered to sustain training throughput.

##### Intra-node bandwidth.

Since RTX 5090 does not provide NVLink, the only intra-node interconnect is PCIe. NVIDIA disables PCIe P2P on non-data-center GPUs, which forces GPU communication traffic to be staged through host memory, also known as “ping-pong” transfer, which can substantially reduce communication efficiency. However, this restriction appears to be enforced by the NVIDIA GPU driver rather than the limitation of GPU hardware. We used a modified version of the open-source NVIDIA driver aikitoria [2026] , along with the necessary platform configuration changes Chen [2026b] , including disabling IOMMU and PCIe ACS as well as adjusting NPS (NUMA Per Socket, also known as Sub-NUMA Clustering on some platforms) to enable GPU P2P on our servers.

On our hardware, enabling P2P improved one-way bandwidth from 31.5 GB/s to 56 GB/s. The former is bounded by half of the PCIe 5.0 × \times 16 bandwidth, i.e., 32 GB/s, because ping-pong transfer requires two host-memory copies; the latter approaches the 64 GB/s PCIe link limit. Bidirectional bandwidth increased from 32 GB/s to 111 GB/s, compared with theoretical limits of 128 GB/s. Communication latency decreased drastically from 14.3 μ \mu s to 0.4 μ \mu s. The eight-GPU AllReduce bandwidth over PCIe, measured as busbw reported by nccl-tests , increased from 14.75 GB/s to 27.34 GB/s. When the test was restricted to the four GPUs attached to a single CPU socket, the gain was larger, from 16.33 GB/s to 46.31 GB/s, about 2.8 × \times . Adding a PLX PCIe switch between the GPUs and the root complex might further improve P2P bandwidth toward the link limit; this is a prospective topology change, not a configuration used in the reported run.

##### Inter-node bandwidth.

To scale training beyond a single server, the inter-node network must provide bandwidth comparable to the intra-node communication path. We set up a 400 Gbps InfiniBand network among our servers, i.e., 100 GB/s bidirectional bandwidth: one dual-port Mellanox ConnectX-7 host card adapter (HCA) is installed in each server, with both ports connected to a 200 Gbps InfiniBand HDR switch. Half of the GPUs in the server could communicate with the HCA directly, while the other half must route through the inter-socket link to reach the HCA.

Even after PCIe P2P is enabled, non-data-center NVIDIA GPUs still do not support GPUDirect RDMA (shortened as GDR), another performance-critical feature for inter-node communication. GDR builds on P2P and allows a GPU to access remote GPU memory over RDMA NICs without staging through host memory. Through testing, we found that this restriction is also software-enforced: a minimal binary modification to the CUDA user-space driver can enable GDR on RTX 5090 Chen [2026a] . Due to EULA constraints, we are not able to disclose the details in this report. When this configuration is unavailable, the same training recipe can be run with the stock NVIDIA driver, falling back to the standard communication path at lower inter-node bandwidth. In end-to-end tests, enabling GDR improved the 24-GPU (3 servers) AllReduce bandwidth, measured as busbw , from approximately 8.87 GB/s to 19.93 GB/s. We believe that if a more balanced network topology is used (e.g., one InfiniBand HCA on each socket), the performance gain could be even higher.

##### Caveats.

These modifications should be used with caution. First, modifying drivers is unsupported by the vendor and must be done at the user’s own risk. Second, these changes cannot exceed the physical PCIe bandwidth limit. Third, P2P and GDR should not be enabled blindly; the optimal setting depends on the actual hardware topology and platform capability. For example, if the PCIe root complex of a CPU has limited peer-switching capability (which is often the case on low-end models), heavy P2P traffic among many devices may become congested, and host-memory ping-pong transfer can be faster.

#### 3.1.3 Efficient Training System

We build our training system on Megatron Core Shoeybi et al. [2019] , specifically release v0.16.0 NVIDIA Corporation [2026a] , together with its Transformer Engine dependency. This software stack provides native support for blockwise FP8 training and the Muon optimizer, making it a suitable basis for a model at our scale. Nevertheless, the combination of consumer GPUs, a mixed-precision training recipe, and our model architecture still requires careful system-level tuning.

##### Communication-aware parallelism.

The limited communication bandwidth of RTX 5090 motivates us to use only parallel dimensions with modest communication demands. We combine data parallelism (DP), whose dominant collectives are gradient ReduceScatter and parameter AllGather around optimization, with pipeline parallelism (PP), which exchanges hidden-state tensors only at stage boundaries Narayanan et al. [2021] . We do not use tensor parallelism (TP), which introduces frequent collectives within every Transformer layer. Compared with data-center-class systems, our communication disadvantage is substantially more pronounced within a node. We therefore depart from Megatron Core’s default rank ordering and use a pp-dp ordering that places each PP group on topologically closer GPUs within a node, thereby reducing sensitivity to communication latency.

##### Appropriate micro-batch size.