Puro-2B: Poor Lab's Qwen2-1.5B Trained on RTX 5090 within $5090 — part 4/7

Cleaned-text code points [46994, 62401). Bibliography omitted here; source.md includes it.

In the selected 170M diagnostic, MuonH and effective-LR aligned Muon converge more slowly at the beginning but outperform ordinary Muon near the end of training. This pattern is consistent with the learning rate analysis in Appendix F . In particular, the MPL fit [ Luo et al., 2025b ] provides a simple diagnostic relationship: a decrease in learning rate can induce an approximately proportional reduction in loss, with the loss decrease taking effect over subsequent training steps. Applying this view to the effective LR, ordinary Muon decays its effective LR more aggressively early in training, which is consistent with its faster early loss reduction but leaves less effective LR decay for the later stage. MuonH and aligned Muon instead follow a more gradual, approximately linear effective LR decay, giving up some early convergence speed while retaining more decay toward the end of training. This view provides additional motivation for using a linear-decay schedule in our production design.

Together, these diagnostics motivate treating effective LR as the primary hyperparameter for MuonH-wrapped weights. They also suggest that the placement of its decay can affect where loss reduction occurs during training. We therefore next study the practical design of the LR schedule for production run.

#### 3.3.2 Learning Rate Schedule Design

##### Continual training motivates an open-ended schedule followed by a controlled terminal decay.

In practice, the total token budget is commonly undetermined in the beginning [ Hu et al., 2024 , Shen et al., 2024b , Allal et al., 2025 ] . Hence, continual pretraining is a practical setting and may change the data mixture after the initial horizon. We therefore want to retain an open-ended schedule before applying a controlled terminal decay. Warmup-Stable-Decay (WSD) schedule provides a testbed for choosing the decay duration [ Hu et al., 2024 ] . Motivated by the previous section’s experiments, we use a linear decay function in WSD. All sweeps use a 0.6 0.6 B model with the Qwen3-0.6B architecture and global batch size 512. At 20 tokens per parameter (TPP), or 11.3 11.3 B tokens, we vary the effective peak from 0.008 0.008 to 0.024 0.024 in increments of 0.004 0.004 and the WSD decay ratio from 0.2 0.2 to 1.0 1.0 in increments of 0.2 0.2 . At peaks 0.008 0.008 and 0.012 0.012 , we additionally test TPP 50 and 100.

##### Observation 1: a larger peak LR calls for a longer decay.

In the left panel of Figure 6 , the best grid points at effective peaks 0.008 0.008 , 0.012 0.012 , 0.016 0.016 , 0.020 0.020 , and 0.024 0.024 are decay ratios 0.4 0.4 , 1.0 1.0 , 0.8 0.8 , 0.8 0.8 , and 1.0 1.0 . These point optima fluctuate because the long-decay region is broad. A more stable summary is the lower edge of the region within 0.01 0.01 validation loss of the minimum: it moves nondecreasingly as 0.4 0.4 , 0.4 0.4 , 0.6 0.6 , 0.6 0.6 , and 0.8 0.8 , as shown in Figure 18 . At peaks 0.020 0.020 and 0.024 0.024 , ratios 0.8 0.8 and 1.0 1.0 differ by only 0.0005 0.0005 and 0.0044 0.0044 loss. Thus an almost fully linear decay is already competitive near the high end of the tested peak range. The two-anchor interpolation in Figure 17 provides a complementary check: using only ratios 0.2 0.2 and 1.0 1.0 at peak 0.024 0.024 , it places the estimated minimum at ratio 0.85 0.85 within the same long-decay region.

##### Observation 2: a longer training horizon also calls for a longer decay.

At effective peak 0.012 0.012 , the center panel of Figure 6 shows the competitive region shifting toward longer decay. Its lower edge moves from 0.4 0.4 at TPP 20 to 0.6 0.6 at TPP 50 and 100, while the point optima are 1.0 1.0 , 0.8 0.8 , and 0.8 0.8 . At peak 0.008 0.008 , the lower edge remains 0.4 0.4 , although the point optimum moves from 0.4 0.4 to 0.6 0.6 . The latter TPP100 grid lacks ratio 0.2 0.2 , so we treat it only as supporting evidence. The complete peak- 0.012 0.012 lower-edge trace appears in Figure 18 ; the conclusion is a broad shift toward longer decay, not a precise monotone point optimum.

##### The production schedule combines these three practical features.

The right panel of Figure 6 shows the complete production effective-LR schedule. Here, we present the Hyperball weight LR, which is numerically the effective LR for wrapped matrices, instead of the base LR. Phase 1 uses the open-ended power schedule in Equation 10 , decreasing from approximately 5 × 10 − 2 5\times 10^{-2} to 1.04 × 10 − 2 1.04\times 10^{-2} over 438.8B. Its tail preserves a continuation path as the data distribution evolves [ Shen et al., 2024b ] . Phase 2 joins near 10 − 2 10^{-2} and uses a long linear decay over 960.0B rather than spending most of training in a stable segment followed by a short terminal drop. The design therefore combines continual trainability, a high initial effective LR, and a long final decay.

##### Multi-power law provides a back-of-the-envelope, cross-schedule estimator under limited compute.

Before a long run, exhaustively sweeping every decay ratio is impractical. Therefore, we fit a multi-power law (MPL) Luo et al. [2025b] as a diagnostic, using only the two endpoint schedules. The runs are at TPP 20 and with decay ratios 0.2 0.2 and 1.0 1.0 . Two curves per effective peak recover the increasing peak-to-decay trend: the estimated ratio rises from 0.33 0.33 at peak 0.008 0.008 to 0.85 0.85 at peak 0.024 0.024 . Although it is not a certificate of optimality, we use this result as a back-of-the-envelope trend diagnostic. The fitting protocol, unseen schedules, and errors are reported in Appendix F .

### 3.4 Curriculum Model Averaging

When open-source datasets provide usable sample-level scores, we rank samples within each source and construct a mixture-preserving curriculum that places more preferred samples later in training. This ordering is designed to improve the data efficiency of higher-quality samples. However, a conventional terminal learning-rate decay can work against this objective: examples presented near the end of the curriculum are also processed when parameter updates have become small [ Luo et al., 2026 ] . Curriculum Model Averaging (CMA) [ Luo et al., 2026 ] addresses this conflict by combining an ascending data curriculum with constant-LR training and averaging late checkpoints. Our production recipe adopts this principle in Phase 2. We first follow the scheduled learning-rate trajectory, then resume from a selected late checkpoint, hold the base learning rate fixed, and average six checkpoints from the continuation. The complete data and optimization flow is summarized in Figure 7 , with implementation details provided in Sections H.1 and H.3 . We next describe the curriculum construction, the selected constant-LR continuation with checkpoint averaging, and the corresponding ablation results.

For convenience, we label the Phase 2 variants: UD ( U niform data with LR D ecay) globally reshuffles the Phase 2 pool and follows the scheduled learning-rate decay. CD ( C urriculum data with LR D ecay) keeps that decay but presents the data pool in the component-local curriculum order. CDC ( C urriculum data, with LR D ecay then C onstant LR) uses the curriculum order, follows the decay, then continues at a constant learning rate, and reports the final checkpoint. CMA ( C urriculum M odel A veraging) is the production variant: it averages the last several checkpoints from the CDC recipe.

##### The data curriculum arranges the order within each component while keeping the target mixture weight.

We first establish a reproducible order separately for every retained component. If a component provides a usable sample-level quality signal, we sort its examples according to the source-specific interpretation of its score, placing less preferred score ranges earlier and more preferred ranges later. Components without a usable score instead follow a fixed random order. We then assign each example a normalized within-component rank according to its position in this ordered sequence, measured by cumulative token mass. Intuitively, this rank indicates which source-local score range an example falls into. For example, a rank of 0.25 0.25 means that approximately one quarter of the component’s tokens appear earlier in its ordered sequence.

We divide the normalized rank range into 376 intervals, with each curriculum bucket formed by taking the corresponding interval from every component. Each bucket contains approximately 2.5 2.5 B tokens, drawing roughly 1 / 376 1/376 of tokens from every component and preserving the selected cross-component mixture. As training advances through the buckets, scored components move toward their more preferred score ranges, while unscored components advance through their fixed random orders. Since scores are interpreted only within their own sources and are not compared across components, this construction is not a global quality ranking. The scalable approximation used to construct the 376 buckets and the exact transition schedule are given in Sections H.1 and H.2 and Table 19 .

##### The canonical CMA run uses the best-performing late-stage continuation with model averaging.

We compare three late-stage training strategies: averaging checkpoints directly from the terminal-decay trajectory, and two constant-LR continuations resumed from steps 215,000 and 218,000. Among these configurations, the continuation from step 218,000 gives the strongest observed performance and is therefore selected for the production model. For this continuation, training resumes from step 218,000 with the base learning rate fixed at 4.08 × 10 − 5 4.08\times 10^{-5} . With the production Hyperball multiplier m = 10 m=10 , the Hyperball weight LR is therefore 4.08 × 10 − 4 4.08\times 10^{-4} , which is also the effective LR for the MuonH-wrapped matrices. During the late stage of this continuation, we save checkpoints at approximately regular intervals and form the released model by equally averaging the parameters of six such checkpoints. Neighboring checkpoints are typically separated by 100 optimizer steps, or approximately 0.63 0.63 B tokens under the production batch configuration. Full resume, checkpoint-spacing, and averaging details are provided in Section H.3 .

##### Ablation study on curriculum, model average, and const-LR continuation.

The endpoint comparisons in Figure 13(b) support a structured comparison of the jointly selected recipe ingredients, as illustrated in Appendices G and 13(b) . First, curriculum ordering improves over uniform ordering by 1.18 1.18 points without model averaging (CD: 57.17 versus UD: 55.99) and by 1.61 1.61 points when comparing the model averages of the corresponding decay trajectories (CD model average: 57.18 versus UD model average: 55.57). Second, model averaging is not independently beneficial in both ordering conditions : it changes the UD endpoint by − 0.42 -0.42 points and the CD endpoint by + 0.01 +0.01 points without constant-LR continuation. Finally, the two CDC controls score 55.64 and 57.12 for the 215k and 218k continuations, respectively, when evaluated at their unaveraged final checkpoints. With six-checkpoint averaging, the corresponding branches reach 56.80 and 57.81. We therefore select the 218k averaged branch as the production CMA endpoint. It combines curriculum ordering, late constant-LR continuation and six-checkpoint averaging. Part of the observed advantage of CD over UD ordering could be attributed to benchmark contamination in the later-stage data, as discussed in Appendix A . But the remaining advantage is observed after post-training in Section 4.4 , and the contamination alone is unlikely to explain the persistent improvement. We report the exact base LR and Hyperball effective LR, checkpoint IDs, and the model-export procedure in Section H.3 .

### 3.5 Post-Training Recipe

##### Motivation and question.

In the above, we present a complete, low-cost pretraining recipe. The resulting pretrained checkpoints are later used for post-training. Here we focus on supervised fine-tuning (SFT). This raises a natural but underexplored question: does the choice of pretraining recipe affect model performance after SFT? A released checkpoint alone is usually not enough to answer this question. Even when a checkpoint is publicly available, its full training recipe and data may be unavailable. This makes it difficult and expensive to rerun the training and ablate specific pretraining choices. Small-scale ablations are easier, but the undertrained base models may be too weak to show a stable post-training performance [ Qi et al., 2025 ] . Therefore, our low-cost, fully open recipe allows us to study this question at the scale of the final model.

In particular, we compare the SFT results of checkpoints from two Phase 2 recipes: uniform data ordering with learning rate decay (UD) and Curriculum Model Averaging (CMA). The UD checkpoint performs worse on the base-model benchmark suite than the CMA checkpoint. Although both recipes use the same model architecture and Phase 2 training corpus, they differ in data ordering and checkpoint construction. This comparison therefore requires two separate Phase 2 training runs. In the CMA recipe, the curriculum places more high-quality data near the end of training. This changes the data distribution over the course of Phase 2. The base-model improvement may therefore be temporary and tied mainly to the final stage of pretraining. We test whether this improvement remains after both checkpoints undergo the same SFT procedure.

##### Data construction and experimental setup.

To study this question, we apply the same SFT procedure to the UD and CMA checkpoints for each data setup. All other training conditions are matched within the same setup. We continue to use the MuonH optimizer, as in pretraining [ Wen et al., 2026 ] . The base learning rate follows a cosine schedule that decays from 1 × 10 − 5 1\times 10^{-5} to 1 × 10 − 7 1\times 10^{-7} . The MuonH-managed matrix parameters use a 10 × 10\times multiplier to obtain their Hyperball weight learning rate. We set the global batch size to 160.

To examine the results under different SFT data distributions, we construct three data setups. (1) GSM8K-based SFT . It combines the original GSM8K training split with GSM8K-related synthetic data from MetaMathQA and OpenMathInstruct-2. It also includes selected Tulu-3 components, which cover mathematics and instruction following. (2) Math&Code SFT with replay . It contains a larger collection of mathematics data from GSM8K, MetaMathQA, OpenMathInstruct-1, and OpenMathInstruct-2. It also includes code data from OpenCodeInstruct and Magicoder. We add a small, shuffled replay set from the final part of the Phase 2 curriculum, which contains the highest-quality Phase 2 data. (3) Tulu-3 mixed-domain SFT . It uses the English portion of Tulu-3-sft-mixture-0225 [ Lambert et al., 2024 ] . Because this mixture covers several domains, it provides a broader test of general capability and instruction following.

For all three setups, we deduplicate the training examples and discard empty sequences. For decontamination, we remove any training example that shares a 13-gram with an evaluation test example. Detailed data composition and training settings are provided in Appendix E .

##### Evaluation.