The Loss Does Not See the Basis, but Adam Does — part 11/12

Cleaned-text code points [153649, 169622). Bibliography omitted here; source.md includes it.

The mechanism-identification experiments presented in the main text are intentionally compact and deterministic. To directly assess how results vary with scale, we evaluated the full suite of methods on the sensing task across problem sizes n ∈ { 64 , 96 , 128 , 192 , 256 } n\in\{64,96,128,192,256\} (with planted rank 3 3 , m = 2 ​ dof m=2\,\mathrm{dof} , using float64 precision on H100 GPUs), using 10 10 random seeds per configuration. For each method, three seeds are used to choose the learning rate based on the best average recovery performance among the interpolating rates; see Appendix D.3 , where this selection criterion is rigorously tested. The other seven seeds are used only with the selected rate and not in selection; averages presented here are over all 10 10 seeds. Importantly, the seven holdout seeds alone recapitulate the same two-cluster ranking order at each problem size; e.g., at n = 256 n{=}256 , max ⁡ ( GD , sc-Adam , Muon ) = 0.56 < 0.71 = min ⁡ ( clean coordinate-wise ) \max(\mathrm{GD},\text{sc-Adam},\mathrm{Muon}){=}0.56<0.71{=}\min(\text{clean coordinate-wise}) vs. 0.55 < 0.71 0.55<0.71 using all the seeds in Table 11 (in which “sc-Adam” means scalar-Adam and “clean” means excluding the flagged signum ‡ ). The training budget is fixed at 4 × 10 4 4\times 10^{4} steps.

Learning rate grids for each problem size are scaled so that each method operates within the interpolating regime across all sizes: GD scales by 40 / n 40/n ; Muon, Shampoo, and ScaledGD by 40 / n \sqrt{40/n} , with ScaledGD included as a schedule control but not reported; Adam-family methods scale by n / 40 \sqrt{n/40} . At n = 40 n{=}40 , these grids match exactly those used in the main paper, serving as a validation check. A cosine-annealing control was also run at each size. All methods yield nearly identical recovery values under both the constant-step and cosine schedules—up to three decimal places—with one exception: RMSProp, whose selected rate becomes unstable at n = 64 n{=}64 and n = 96 n{=}96 (recovery shifts from 0.596 0.596 to 0.610 0.610 and 0.615 0.615 to 0.767 0.767 under cosine), though it remains within the coordinate-wise cluster regardless. Thus, the relative ordering in the ladder is unaffected by the choice of schedule. The ladder stops at n = 256 n{=}256 : n = 384 n{=}384 is omitted because boundary effects make the grid uninformative there, with Muon no longer interpolating.

† Shampoo’s damping λ = 1 \lambda{=}1 is fixed independently of problem size, which is why it does poorly at large n n —a hyperparameter choice, not an intrinsic limitation. ‡ signum reaches comparable performance on a sufficiently long training schedule, through a different mechanism (Appendix D.10 ). § RMSProp has one outlier at n = 96 n{=}96 , where a poorly chosen learning rate makes it diverge on some seeds.

##### Discussion around the ladder.

The ranking in Table 11 is what underlies our conclusion, and one number in it is easy to misread: the Adam-family ratio with respect to GD decreases with growing n n , but this is not because Adam is relatively better, rather because GD’s absolute value climbs toward its flow-limit value under the same step budget. Both GD and scalar-Adam would approach the continuous-time flow limit given smaller steps and more of them, which is why the ordering rather than the absolute error is what the table is asked to carry.

The n = 40 n{=}40 canary run correctly recovers the results of the paper when the protocol is followed, giving 0.424 0.424 for Lion (versus 0.425 0.425 in the paper) and 0.543 0.543 for Adafactor (matching the paper’s 0.543 0.543 ), while Adam stays flat across the extended grid, registering 0.550 0.550 at its finest interpolating rate. Meanwhile, both GD and scalar-Adam achieve better recovery with smaller step sizes, reaching 0.108 0.108 and 0.143 0.143 respectively on their finest grids (GD’s 0.108 0.108 cell stops at a mean training loss of 1.1 × 10 − 7 1.1\times 10^{-7} , just above the bar, so we read it as a flow-limit value rather than an interpolating one)—agreeing with the flow-limit behavior described in Remark A.5 , which is why the larger- n n values on the ladder should be considered conservative. In particular, two cases were kept in the table for transparency rather than optimized out: Shampoo with untuned damping, and signum’s long-horizon recovery. Neither affects the stated ranking, and both are flagged for further investigation (§ 11 ).

### D.10 Recovery without equivariance: annealed sign descent at long horizons

The zoo’s two-cluster separation (§ 5 ) is defined under a fixed computational budget. However, one coordinate-wise method manages to escape the collapsed cluster when training is extended: signum (signSGD with momentum) using cosine decay over a prolonged annealing schedule. At the paper’s standard budget of 2 × 10 4 2\times 10^{4} steps, its recovery score is 0.445 0.445 —classified as collapsed. When extended to the ladder’s 4 × 10 4 4\times 10^{4} -step horizon, recovery drops to near-zero (below 10 − 5 10^{-5} ) for all n ≥ 64 n\geq 64 (across all 10 10 random seeds). The transition begins at n = 40 n{=}40 , a borderline scenario depending on numerical precision and the random seed. On H100 in float64 , two of the three seeds recover, for a mean error of 0.013 0.013 ; the original CPU-based implementation ( optimizer_zoo_bias.py , MAX_STEPS = 4 × 10 4 {=}4\times 10^{4} , lr 3 × 10 − 3 3\times 10^{-3} ), meanwhile, gives a mean of 0.054 0.054 , with the run using seed 42 42 recovering to 1.5 × 10 − 5 1.5\times 10^{-5} ; and at lr 10 − 3 10^{-3} , recovery stays poor at about 0.71 0.71 throughout.

The values shift between individual runs and platforms, but the configuration as a whole—large step sizes and an extended annealing schedule—reproduces the effect. It is not random, nor is it due to numerical instability: we have observed the phenomenon but do not yet have a mechanistic account of it, so we report it as an empirical fact.

The example does not disprove the theorem, but instead demonstrates where it does not apply. Theorem 4.6 concerns flows with memoryless updates and a scalar preconditioner shared across the factors, and does not consider alternatives. It is entirely possible that a different set of update rules, not respecting the equivariance of the loss under the gauge, would have different dynamics and still be able to exploit the low-rank structure of the target signal. We considered one possibility and ruled it out.

A possible explanation could be that signum’s momentum has an intrinsic rank-one structure: if the momentum were an outer product, then since sign ⁡ ( σ ​ u ​ v ⊤ ) = sign ⁡ ( u ) ​ sign ​ ( v ) ⊤ \mathrm{sign}(\sigma uv^{\top})=\mathrm{sign}(u)\mathrm{sign}(v)^{\top} for σ > 0 \sigma>0 , the signed update would itself be rank-one and aligned with the planted signal. This does not appear to be the case. At 4 × 10 4 4\times 10^{4} steps, the successful runs at lr 3 × 10 − 3 3\times 10^{-3} have a ratio of the top two singular values of the momentum of σ 1 / σ 2 ≈ 1.1 \sigma_{1}/\sigma_{2}\approx 1.1 – 1.3 1.3 at convergence, and the signed updates themselves have even lower values ( 1.05 1.05 – 1.17 1.17 ). The fraction of squared Frobenius norm contained in the leading singular component—the top-mode energy—lies between 10 % 10\% and 24 % 24\% for these runs ( experiments/signum_c9_audit.py ), against the 100 % 100\% a genuinely rank-one matrix would carry. The leading mode therefore holds only a small share of the update’s energy, so the update as a whole cannot have been rank-one. The mechanism by which signum achieves lower-rank recovery after long annealing schedules remains an open question.

A more plausible theory concerns the ℓ ∞ \ell_{\infty} or steepest-descent nature of sign-based updates ( Bernstein et al., 2018 ; Xie & Li, 2024 ) , which implicitly biases towards max-margin solutions— Baek et al. (2026) prove that signum converges to the ℓ ∞ \ell_{\infty} -max-margin classifier at any batch size. These, in turn, have low rank only if the ground truth is also low-rank, which would explain the collapse of the corner in the presence of any nontrivial spectral tail (as illustrated below); we posit this as an open problem.

Finally, this example differs from the common-scalar case in that it does not appear to stem from a continuous limit. First, it emerges from large learning rates and long annealing schedules rather than from a smooth flow limit. Specifically, at a learning rate of 3 × 10 − 3 3\times 10^{-3} , the average recovery over three random seeds improves from 0.445 → 0.054 → 0.000 0.445\to 0.054\to 0.000 as the cosine-annealing schedule is extended from 2 × 10 4 → 4 × 10 4 → 6 × 10 4 2\times 10^{4}\to 4\times 10^{4}\to 6\times 10^{4} steps. By contrast, at a learning rate of 10 − 3 10^{-3} , it remains poor ( 0.65 0.65 – 0.72 0.72 ) throughout the schedule. A linear decay schedule has a similar threshold behavior ( 0.479 → 0.103 → 0.000 0.479\to 0.103\to 0.000 at analogous durations), which suggests that the issue is not specific to cosine annealing.

Second, the corner structure breaks down when a spectral tail is added to the planted matrix. In the tail sweep described in § 8 , signum at lr 3 × 10 − 3 3\times 10^{-3} and 4 × 10 4 4\times 10^{4} steps shows recovery values of 0.025 0.025 , 0.138 0.138 , 0.280 0.280 , 0.422 0.422 , 0.646 0.646 , and 0.796 0.796 at τ = 0 \tau=0 , 0.05 0.05 , 0.10 0.10 , 0.20 0.20 , 0.35 0.35 , and 0.50 0.50 respectively (averaged over three seeds; the τ = 0 \tau{=}0 value again reflects the seed-sensitive borderline case, and sign descent is chaotic enough here that these digits shift by up to 0.03 0.03 across platforms while the monotone breakdown does not). Meanwhile, common-scalar methods follow the smoother degradation patterns seen in Table 4 . Thus, we view signum as an exceptional case that helps define boundaries: non-equivariant dynamics can achieve low-rank solutions only in a narrow regime—exactly low-rank settings, large steps, and long anneals—while common-scalar equivariance supports a more robust, generalizable, and analyzable gradient-flow mechanism.

## Appendix E Extended related work

##### Transformation invariance in optimizer design.

The engineering literature has grappled with similar design choices at a lower level; for instance, Ling et al. (2022) observe that for vector-valued geometry parameters Adam’s per-coordinate moment estimation incorrectly violates rotation equivariance, and restore it by using a single shared scalar per vector. This is an early optimizer-level analog of the continuous transformations that our dial (§ 7 ) performs. Recently, LoRA-RITE ( Yen et al., 2025 ) proposes a transformation-invariant matrix preconditioner on LoRA (low-rank adaptation, Hu et al. 2022 ) parameters, which improves practical efficiency by explicitly accounting for non-invariance as a source of optimization error. We import this line of reasoning to the implicit-bias setting, showing that failure to respect invariance not only slows down training but biases the solution, in the factored parameterization that a LoRA adapter instantiates (§ 5 ) and in standard attention layers (§ 6 ). Riemannian and preconditioned optimizers such as ScaledGD ( Tong et al., 2021 ) incorporate similar domain knowledge for better conditioning; ScaledGD is also our within-class counterexample (Remark 4.7 ).

Independently, Zhang (2026) discovers a similar local phenomenon in a different application domain: in group-equivariant networks, Muon’s polar step respects orthogonal transformations in the Schur multiplicity basis while AdamW’s coordinate-wise preconditioning does not; this leads to a new optimizer, PolarAdamW, which deliberately breaks equivariance to obtain per-coordinate adaptation. These works share the starting point with us in that they identify a local covariance structure that should be respected by the optimizer, but we are, to our knowledge, the first to analyze its consequences for solutions recovered by factored models. Their observation that coordinate-wise preconditioning is most useful when the multiplicity gauge is trivial corresponds to our boundary analysis (§ 10.1 ).

Recent works explore similar themes from different angles: Lau & Su (2026) propose that update directions should respect the symmetry group of each weight block, unify spectral descent, Muon, Scion, and their polar variants, and derive equivariant optimizers for permutation and shift symmetry. Shirodkar (2026) generalizes standard optimizers to be exactly G G -equivariant across architectural gauges, e.g., rotations within each attention head; the outcomes it reports—an Adam variant that resists the over-training collapse AdamW falls into, and a Muon variant that groks at a depth plain Muon never reaches—are selection effects of the kind our mechanism predicts, arrived at from the design side. Stupariu & Manolache (2026) directly compare Muon and Adam in equivariant settings. We view these works as providing important context and design inspiration; in particular, by analyzing the interaction between optimization and symmetry we highlight cases where breaking equivariance (§ 10.1 ) is desirable.

##### Symmetry-driven accounts of implicit bias.

Accounts based on symmetry offer a framework for understanding implicit bias. Aladrah et al. (2026) propose a complementary stochastic mechanism in which continuous predictor-preserving symmetries interact with noise from stochastic gradient descent (SGD), leading to a geometric correction—based on quotient volume—to the effective loss. This adjustment accounts for balancing and spectral effects and enables reverse engineering of the bias. Their illustrative case involves positive rescaling symmetry ( U ​ D , V ​ D − 1 ) (UD,VD^{-1}) . In contrast, our approach is deterministic and tied to specific optimizers rather than driven by noise or applicable universally across them: full-batch update rules either commute with or explicitly depend on the orthogonal gauge, and this property alone determines which interpolant is selected under a fixed computational budget (as verified in our bitwise-deterministic twin experiments). These two mechanisms are treated as separate in our analysis, and how they might interact under mini-batch training remains an open question noted in § 11 .

##### Muon and structured optimizers.

Regarding structured optimizers, Muon ( Jordan et al., 2024 ; Liu et al., 2025 ) applies matrix-sign orthogonalization to momentum, while Shampoo ( Gupta et al., 2018 ) uses Kronecker-factored preconditioning—both methods seeing practical use. Recent theoretical work attributes a spectral max-margin bias to Muon ( Fan et al., 2025 ; Gronich & Vardi, 2026 ) , and Kang et al. (2026) establish equal-rate singular-value growth for Muon when applied to LoRA-style factorizations. Our phase diagram captures these dynamics, predicting when Muon’s scheduling is beneficial (for exactly low-rank targets) versus detrimental (for targets with heavy spectral tails), aligning with the dual empirical behavior observed by Dragutinović et al. (2026) .

Independently, Beneventano et al. (2026) identify the polar step as the entropy-maximizing (spectrum-flattening) choice in one step and derive exact singular-value dynamics in underdetermined regression, providing theoretical grounding for the tail-sensitive mechanism measured in our phase diagram. Shen et al. (2026) arrive at a consistent view regarding performance trade-offs in mixed-spectrum sensing tasks. Dong & Sawin (2026) directly implement a within-class family by raising gradient singular values to a fractional power, effectively interpolating between Muon and gradient descent along the spectral-transfer axis identified in our structure theorem (§ 4 ) and visualized in our phase diagram. Our two axes place all of these on one map.