Puro-2B: Poor Lab's Qwen2-1.5B Trained on RTX 5090 within $5090 — part 7/7

Cleaned-text code points [93560, 98539). Bibliography omitted here; source.md includes it.

Low-precision training primarily reduces the cost of executing a fixed workload. FP8 mixed precision accelerates the matrix multiplications that dominate Transformer training while retaining higher precision where needed. DeepSeek-V3 shows that this design can remain stable at large scale [ Micikevicius et al., 2022 , DeepSeek-AI et al., 2024 ] . Quartet II extends this direction to NVFP4 and improves its numerical behavior [ Panferov et al., 2026 ] . These methods improve execution efficiency for a given training workload, but do not by themselves reduce the token or nominal FLOP budget required to reach a target quality.

A complementary line of work reduces the training workload through model architecture, data selection, staged training, and systems optimization. TinyLlama, JetMoE, Yulan-Mini-2.4B, and Kaiyuan-2B explore different combinations of these techniques [ Zhang et al., 2024a , Shen et al., 2024a , Yiwen et al., 2025 , Luo et al., 2025a ] . Collectively, these projects show that useful small-model capability can be obtained under substantially smaller compute budgets than those of large-scale foundation models. The reported runs, however, generally rely on specialized data-center accelerators, so the hardware barrier for a small research group remains.

Hardware accessibility introduces a separate constraint. QLoRA reduces the memory required to fine-tune a pretrained model, leaving the original pretraining cost unchanged [ Dettmers et al., 2023 ] . LLMQ studies full-model pretraining on RTX 4090 GPUs using FP8, recomputation, sharding, and CPU offloading [ Schultheis and Alistarh, 2025 ] , while Quartet II measures the throughput benefit of NVFP4 on a single RTX 5090 [ Panferov et al., 2026 ] . The studies considered here establish the feasibility of relevant training operations on consumer GPUs, but focus primarily on systems feasibility or throughput rather than on releasing a trillion-token-scale base model with an explicit replication-cost boundary.

Puro-2B lies at the intersection of these directions. It is a from-scratch 2B base model trained on more than 1.4T tokens using FP8 on a multi-node RTX 5090 cluster, with measured accelerator usage and an explicit replication-cost boundary. We therefore evaluate the training recipe and hardware platform end to end, measuring both the resulting model quality and replication cost. Together, these components provide researchers with a reproducible and cost-accessible end-to-end pipeline for studying pretraining behavior and testing alternative methods at meaningful scales.

## 6 Conclusion and Future Direction

In this report, we present a cost-efficient, hardware-accessible reproducibility recipe for language model pretraining, and use it to train the Puro-2B model collection from scratch on consumer-grade RTX 5090 GPUs. Across different training budgets and recipe variants, the collection provides a direct view of the cost–performance tradeoff of our pipeline. A model from this collection reaches the performance of Qwen2-1.5B at about $4.4K, while our best checkpoint approaches Qwen2.5-1.5B at the canonical accelerator-only cost of about $6.9K. Based on these runs, we further formulate the Puro Cost Scaling Law to characterize how model capability scales down with training cost under our recipe. More broadly, these results demonstrate that useful from-scratch pretraining at the billion-parameter scale is feasible with modest budgets and accessible hardware.

There remain several directions for extending this foundation. First, we plan to extend the reproducibility recipe beyond pretraining to post-training, with particular emphasis on developing and studying agentic capabilities. Second, we plan to broaden the architectural space beyond standard dense Transformers, including looped Transformers, linear-attention models, mixture-of-experts architectures, and other emerging designs. Third, we aim to expand the hardware recipes beyond RTX 5090 GPUs and cover a wider range of training budgets, making it easier to study how the optimal system and training choices change across hardware and scale. We view Puro-2B not as an endpoint, but as evidence that affordable and inspectable model training is practical today, and as a baseline for future efforts to push the frontier toward lower cost, broader accessibility, and higher efficiency.

## 7 Acknowledgments

The authors would like to thank Yanfu Investments for providing computational resources. Zhenbo Sun and Chenyi Dang contributed during the early stages of this work. The authors also thank Kaiyue Wen, Kexian Tang, Minxing Yang, Zhan Ling, Haodong Wen, Shaowen Wang, Huaqing Zhang and Shuo Huang for their valuable feedback and insightful discussions, and Yanzheng Cai, Yuanwei Wang, Zhixuan Pan, Rui Chen, and Nuo Chen for their assistance with proofreading and manuscript revision. Multiple LLM-based tools were used to assist with drafting, proofreading, and language refinement.