Puro-2B: Poor Lab's Qwen2-1.5B Trained on RTX 5090 within $5090

arxiv:2608.27370

Original source · Paper JSON

Data, training, optimization · 2026-08-27

Cites (0) · Cited by (0)

Complete cleaned Markdown · Source Markdown including bibliography

Read in parts

1. Part 1
2. Part 2
3. Part 3
4. Part 4
5. Part 5
6. Part 6
7. Part 7

Contents

• Puro-2B : Poor Lab’s Qwen2-1.5B Trained on RTX 5090 within $5090 Thanks: Corresponding authors.
• 摘要
• 1 Introduction
• 2 Overview
• 2.1 Pipeline at a Glance
• Model architecture and pretraining system ( Sections 3.1 and 3.2 ).
• Two-phase pretraining, curriculum, and optimization ( Sections 3.3 and 3.4 ).
• Data acquisition, filtering, and mixture ( Section 3.6 ).
• Evaluation and post-training ( Sections 3.5 and 4 ).
• 2.2 Reproduction-Cost Boundary
• 3 Training Recipe
• 3.1 Training Infrastructure
• 3.1.1 RTX 5090 as a Cost-Effective GPU Choice
• 3.1.2 Hardware Setup and Tweaks
• Intra-node bandwidth.
• Inter-node bandwidth.
• Caveats.
• 3.1.3 Efficient Training System
• Communication-aware parallelism.
• Appropriate micro-batch size.
• Overcoming load imbalance.
• 3.2 FP8 Mixed-Precision Training
• 3.3 Hyperball Optimization
• 3.3.1 Effective Learning Rate
• Scale invariance motivates the effective learning rate.
• MuonH makes the effective LR schedule explicit.
• Matching the effective LR schedule also aligns validation loss.
• Effective LR provides a more informative descriptor of the loss trend.
• Effective LR MPL helps interpret the late-stage crossover.
• 3.3.2 Learning Rate Schedule Design
• Continual training motivates an open-ended schedule followed by a controlled terminal decay.
• Observation 1: a larger peak LR calls for a longer decay.
• Observation 2: a longer training horizon also calls for a longer decay.
• The production schedule combines these three practical features.
• Multi-power law provides a back-of-the-envelope, cross-schedule estimator under limited compute.
• 3.4 Curriculum Model Averaging
• The data curriculum arranges the order within each component while keeping the target mixture weight.
• The canonical CMA run uses the best-performing late-stage continuation with model averaging.
• Ablation study on curriculum, model average, and const-LR continuation.
• 3.5 Post-Training Recipe
• Motivation and question.
• Data construction and experimental setup.
• Evaluation.
• 3.6 Data Recipe
• 3.6.1 How Do We Select a Good Data Recipe?
• Fine-grained data slice selection.
• Feature-guided data selection.
• Capability balance across domains.
• 3.6.2 How Do We Preprocess Data and Reproduce the Shards?
• Data preprocessing pipeline.
• Kaiyuan-Spark preprocessing framework.
• Two-phase data and transition schedule.
• 4 Evaluation
• 4.1 Evaluation Setup
• 4.1.1 Model Selection
• 4.1.2 Benchmarks
• 4.1.3 Implementation Details
• 4.2 Resulting Performance
• Mathematics and code.
• Reasoning and knowledge.
• 4.3 Cost Estimation
• 4.3.1 Cost-Saving Factors
• RTX 5090 exhibits a high cost-performance ratio and utilization.
• MuonH provides a quality-equivalent compute shift in the matched ladder.
• Blockwise FP8 retains a stable quality fraction while accelerating the 2B proxy.
• The Puro Cost Scaling Law summarizes recipe-specific scale-down behavior.
• 4.3.2 Cross-Model Reproduction Cost
• Estimation protocol.
• Pareto-frontier result.
• 4.4 Post-Training Results and Analysis
• GSM8K-based SFT.
• Math&Code SFT with replay.
• Tulu-3 mixed-domain SFT.
• 5 Related Works
• 5.1 Open-Recipe Language Models
• 5.2 Low-Cost Language Model Pretraining
• 6 Conclusion and Future Direction
• 7 Acknowledgments