The Loss Does Not See the Basis, but Adam Does — part 10/12

Cleaned-text code points [138871, 153649). Bibliography omitted here; source.md includes it.

Three facts from the experiment support this reading. (i) Adam demonstrated almost no dependence of recovery on the learning rate: across all five tested values it varied only from 0.56 0.56 to 0.58 0.58 . These five learning rates span two orders of magnitude and every one of them interpolates, so the range contains no step size at which Adam keeps the low-rank bias. (ii) GD and scalar-Adam perform better at smaller step sizes (recovery improves from 0.131 0.131 to 0.113 0.113 for GD, and from 0.201 0.201 to 0.165 0.165 for scalar-Adam, at their smallest interpolating rates). This aligns with expectations from the common-scalar flow model, though the transfer theorem does not formally guarantee the same behavior for EMA-based scalar-Adam. (iii) Muon displays the reverse trend: it recovers exactly at larger decayed rates ( 0.0000 0.0000 error at learning rates 0.03 0.03 – 0.1 0.1 ), but degrades at smaller ones ( 0.75 0.75 – 0.91 0.91 recovery). On this task, that behavior depends on the learning schedule, and it is consistent with equal-rate spectral growth ( Kang et al., 2026 ) and the sensitivity of finite Newton–Schulz iterations near small singular values. Thus, in this setting, equivariance defines a stable class membership, while the learning schedule and step size affect recovery performance within that class.

### D.3 Learning-rate curves and selection rules

The grids of Table 7 extend to lower rates than the zoo’s selection procedure used, and the → \to fl rates were run specifically to probe the flow limit; they therefore usually lie below the rates selected by the zoo’s best-recovery criterion (§ 5 ), which explains the discrepancy with Table 2 . Each zoo method appears in the table at the rate chosen for it by that criterion (GD 0.131 0.131 at lr 0.01 0.01 , Adam 0.573 0.573 at 0.01 0.01 , scalar-Adam 0.201 0.201 at 0.001 0.001 ; see the grid note in Appendix C ). The reading is the one given in Appendix D.2 : Adam is flat—no learning rate preserves the bias—while GD and scalar-Adam improve steadily as they approach the flow limit, and Muon is driven by its large decaying steps (a schedule-driven bias). The remaining coordinate-wise methods (RMSProp, Lion, Adafactor, and signum) were swept on their own grids but are omitted from the table, as they all fall into the poor-recovery category ( ≥ 0.42 \geq 0.42 ) under both selection criteria, as the stress test below reports.

Selection-rule stress test: with best-recovery selection under the uniform cosine schedule, the classification achieves 9 / 9 9/9 ; with fastest-interpolation selection it achieves 8 / 9 8/9 , with Muon the sole failure at recovery 0.91 0.91 . Thus the fastest-interpolation, bias-blind rule hurts an otherwise equivariant method, which is why Appendix D.2 reports results in terms of capability (can/cannot) with the curves, rather than under any single selection criterion. “Fastest-interpolating” is defined within the fixed horizon of 2 × 10 4 2\times 10^{4} steps: the rate that crosses the 10 − 7 10^{-7} bar in the fewest steps, not the largest rate that eventually interpolates. For Muon that is the smallest grid rate ( 0.003 0.003 , recovery 0.915 0.915 in Table 7 ); hence the two higher rates ( 0.03 0.03 , 0.1 0.1 ), at which it recovers exactly, are not what this rule picks. The coordinate-wise cluster stays ≥ 0.42 \geq 0.42 under both rules.

### D.4 Initialization scale

Table 8 sweeps the initialization scale for the five reference methods.

### D.5 Attention twins across seeds, initialization draws, and noise levels

Using the deterministic CPU setup described in § 6 , we trained six Adam gauge pairs—combining two different initialization seeds with three independent gauge draws. Initial drift for these runs varied from 3.4 3.4 to 5.4 × 10 − 3 5.4\times 10^{-3} , in good agreement with the same 2L/ d = 64 d{=}64 configuration recalculated on GPU in float64 , whose step-1 drift varied from 4.0 4.0 to 5.7 × 10 − 3 5.7\times 10^{-3} (Table 9 ). Final drift in this case ranged between 0.69 0.69 and 0.80 0.80 . The differences in validation accuracy were insignificant, | Δ ​ acc | ≤ 0.0023 |\Delta\mathrm{acc}|\leq 0.0023 , suggesting that both models perform similarly well but converge to two different points on the solution manifold.

In the case of the Adam noise-twin tests performed in the same basis, noise amplitude varied from 10 − 7 10^{-7} to 10 − 5 10^{-5} , and the final drift in this case varied from 1.6 × 10 − 5 1.6\times 10^{-5} to 1.0 × 10 − 3 1.0\times 10^{-3} . The structural ratio, defined as the ratio of the smallest final gauge drift to the final drift of the noise twin carrying 100 100 times more perturbation, exceeds 600 × 600\times , the actual run value being 662 × 662\times . The rounded figure quoted here is the more cautious lower bound, and is the one we claim.

Among the equivariant rows, the SGD and scalar-Adam gauge twins, over two (seed, draw) combinations each, end at no more than 3.7 × 10 − 4 3.7\times 10^{-4} for SGD (its largest) and 4.3 × 10 − 6 4.3\times 10^{-6} for scalar-Adam.

The Muon chaos metric, the mean | log 10 ⁡ ( gauge / noise ) | |\log_{10}(\text{gauge}/\text{noise})| drift ratio past step 100, equals 0.05 0.05 decades for seed 42. Within that tolerance one may deem the gauge twin a perturbation at the floating-point level, and therefore indicative of chaos rather than of structural basis-dependence.

##### Scaling the twin diagnostic (GPU, float64 ).

Table 9 reports analogous values for deeper and wider models, and on real text inputs. The A = I A{=}I determinism twin is exactly 0 0 for every row. Two observations are universal across all depths and widths: the initial Adam gauge deviation, which is about 10 − 3 10^{-3} – 10 − 2 10^{-2} , and the machine-precision “floor” shared by all three equivariant methods, which is 12 12 – 13 13 orders of magnitude below Adam’s value. The “ratio” column reports the “onset ratio”, defined as the smallest Adam gauge split at step 1 divided by the value of the 100 × 100\times -stronger noise twin at that same step. We report the onset, not the final, ratio, because in deep models the end-state statistics cease to be informative about the dynamics, even in fully deterministic, non-minibatch runs.

At 2L/ d = 64 d{=}64 and 4L/ d = 128 d{=}128 , the equivariant twins remain negligible by the end—scalar-Adam at 6.6 × 10 − 15 6.6\times 10^{-15} and 6.0 × 10 − 12 6.0\times 10^{-12} , SGD at 2.5 × 10 − 9 2.5\times 10^{-9} and 1.6 × 10 − 10 1.6\times 10^{-10} —and the final-drift ratio is substantial (exceeding 500 × 500\times for 2L/ d = 64 d{=}64 ; the CPU value mentioned above reflects the 662 × 662\times of the same comparison on that harness). At 6L/ d = 256 d{=}256 , however, the expectation is violated: scalar-Adam’s gauge twins produce a value of 1.8 1.8 and Muon’s 0.58 0.58 , while SGD gives no signal at that scale, never leaving chance accuracy, so its exactly zero drift reports an untrained model rather than a preserved gauge. The noise twin interprets the rest as instability, with scalar-Adam’s noise twin at 1.7 1.7 and Muon’s at 0.83 0.83 , corresponding roughly to where a 10 − 7 10^{-7} perturbation in the same basis would land at those scales. Similarly, the char-LM has scalar-Adam’s gauge and noise twins both at 0.44 0.44 , while Adam is the only method with a discrepancy between its gauge and noise twin at step 1 1 —which is why the step- 1 1 onset, rather than the final state, carries the claim.

### D.6 Hyperspectral learning-rate hygiene

The learning rates in § 9 are chosen via a train-only criterion: the best rate is the one achieving the lowest training loss within the step budget, with ties broken in favor of the one using fewest steps. We used different seeds for selection than for two of the four evaluation seeds. Following this protocol, GD selects learning rate 10 10 at density 0.15 0.15 , which matches the test-informed choice, but with no data leakage—this is the only rate on the grid that can reach training loss 3 × 10 − 5 3\times 10^{-5} in 3 × 10 4 3\times 10^{4} steps (lr 3 3 reaches 3 × 10 − 4 3\times 10^{-4} , lr 1 1 only 3 × 10 − 3 3\times 10^{-3} ).

A per-rate analysis using four different seeds confirms that this is not an edge-of-stability ( Cohen et al., 2021 ) effect: GD with lr 3 3 tracks the same matched-loss trajectory as the lr = 10 {=}10 run—at every comparable loss, the held-out RMSEs differ by at most 10 − 5 10^{-5} (e.g., both have 0.03487 0.03487 at one point and 0.02207 0.02207 later). At its deepest point achieved (train loss ≤ 3 × 10 − 4 \leq 3\times 10^{-4} , all four seeds), GD(lr = 3 {=}3 ) already outperformed Adam by 12 % 12\% , and the gap widens to 44 % 44\% at deeper levels with the higher learning rate. For the two rates tried on GD and the one density, the matched-loss path is therefore independent of the rate (within the rate class), varying only in how far along the path the optimization gets within the step budget.

##### Two limits of this grid.

(i) Our reported improvement depends on the selection criterion, though not the ordering: train-only picks better rates for faster convergence, hence bigger rates for the same budget. In the GPU-replicated run (Table 5 ), whose grid extends to lr 30 30 , this puts GD(lr = 30 {=}30 ) in its rate-invariant region while giving Adam the worst rate on its own grid. Adam’s smallest rate ( 10 − 3 10^{-3} ) achieves held-out RMSE 0.0171 0.0171 at effective rank 17 17 , compared to GD(lr = 30 {=}30 )’s 0.0148 0.0148 at rank 13 13 . So picking the best rate for each method would have improved GD by 13 % 13\% over Adam, compared to the 43 % 43\% found with the train-only selection (also consistent across the four paired seeds in the GPU experiment). We report the train-only selection because using held-out statistics to both select and evaluate a learning rate would reintroduce the very bias this protocol was designed to avoid (Appendix D.2 ). What does not change with the selection criterion, however, is the ordering and rank profile: at every rate on Adam’s grid ( 0.0171 / 17 0.0171/17 , 0.0198 / 20 0.0198/20 , 0.0260 / 27 0.0260/27 ), the GD run selected achieves both a lower held-out error and lower effective rank than the corresponding Adam run.

(ii) Scalar-Adam is undersampled in these experiments: its rate grid { 3 × 10 − 3 , 10 − 2 , 3 × 10 − 2 } \{3{\times}10^{-3},10^{-2},3{\times}10^{-2}\} sits at 3 × 3\times higher rates than Adam and shows no signs of saturating, achieving lower held-out error still at the smallest rate tried ( 0.0207 0.0207 vs 0.0360 0.0360 at the selected rate) and hence does not reproduce the phenomenon. This is entirely consistent with the finding in § 8 —equivariance defines the algorithm class, while spectral responses determine the dynamics within a class. Real-world data has spectral content that makes this second channel relevant, and scalar-Adam’s EMA of first moments (Remark 4.7 ) fails to satisfy the premise of Theorem 4.6 . A similar consideration applies to Shampoo across problem sizes (Appendix D.9 ).

### D.7 Interpolation-bar audits

RMSProp necessitated the formulation of the rule in § 3 based on performance criteria rather than method class, due to its behavior. Under a constant learning rate, it fails to interpolate, plateauing at a training loss of 2.9 × 10 − 4 2.9\times 10^{-4} . This limit is not due to insufficient training, as even with a threefold increase in training budget (up to 6 × 10 4 6\times 10^{4} steps), the lowest achieved loss remains at 2.8 × 10 − 4 2.8\times 10^{-4} . The bottleneck stems from the ϵ \epsilon -bounded denominator in its update rule, which persists regardless of extended training. Evaluating recovery performance at this plateau would mean comparing a non-interpolating, early-halted run with eight interpolating cases—precisely the kind of comparison the evaluation protocol aims to avoid.

However, when using cosine decay—better suited to RMSProp’s optimization dynamics—it successfully interpolates, achieving a training loss of 6.0 × 10 − 8 6.0\times 10^{-8} at a learning rate of 10 − 4 10^{-4} . In this setting, it reaches a recovery score of 0.5266 0.5266 with an effective rank of 12.80 12.80 . This result falls within the coordinate-wise cluster and lies just 0.0004 0.0004 below the 0.5270 0.5270 score of the constant-rate run—measured at its 2.9 × 10 − 4 2.9\times 10^{-4} plateau, since that run never met the interpolation bar, so 0.5270 0.5270 is a recovery at the plateau, not at interpolation. Thus, the learning schedule has minimal impact on final recovery performance despite significant differences in training dynamics; it determines whether RMSProp is measurable, not what it selects. Once we apply decay, all nine methods satisfy the 10 − 7 10^{-7} interpolation threshold, giving rise to a 9 / 9 9/9 classification (Table 2 ). This conclusion holds true when every method follows the same cosine learning-rate schedule (Appendix D.1 ); thus, none of the clustering results presented here depend on a specific method’s scheduling scheme.

Conversely, our evaluation also turned up the opposite failure case. With a constant learning rate, the signum algorithm stagnates at 5 × 10 − 2 5\times 10^{-2} training loss, attaining a recovery score of 0.009 0.009 . If recovery were measured without consideration of interpolation, this early-stopping behavior would be considered evidence in favor of sign descent being able to maintain bias. In contrast, with interpolation (under decay), signum has a recovery of 0.445 0.445 , indicating that the bias was lost, as expected.

### D.8 FlowAdam-p is not an artifact of early stopping

The bias restoration achieved in § 10 with FlowAdam- p = 0 p{=}0 (using RMS convention, global norm clipping at c = 10 c{=}10 , and learning rate 10 − 3 10^{-3} ) is subject to the same early-stopping ambiguity as any recovery measurement. In order to investigate this, we boosted the training budget to 1.2 × 10 5 1.2\times 10^{5} steps and measured recovery at matched training loss points ranging from 10 − 4 10^{-4} to the stricter 10 − 7 10^{-7} interpolation threshold. Recovery stays relatively flat within this range (Table 10 ): each individual run varies by no more than 3 × 10 − 4 3\times 10^{-4} between 10 − 4 10^{-4} and 10 − 7 10^{-7} , and the values are stable to four decimal places beginning at 10 − 6 10^{-6} . Therefore, the recovery and effective rank obtained with a lax loss floor are nearly indistinguishable to three significant figures from those obtained at full interpolation. The bias is well formed before the final stop, demonstrating that it is not an artifact of the stopping rule.

### D.9 The H100 replication ladder