The Loss Does Not See the Basis, but Adam Does — part 9/12

Cleaned-text code points [123172, 138871). Bibliography omitted here; source.md includes it.

(i) The h ⁡ ( σ ) = η ​ σ h(\sigma)=\eta\sigma (gradient) flow is u ˙ = − η ​ v ​ ( w − s ) \dot{u}=-\eta\,v(w-s) , v ˙ = − η ​ u ​ ( w − s ) \dot{v}=-\eta\,u(w-s) , hence w ˙ = u ˙ ​ v + u ​ v ˙ = − η ⁡ ( u 2 + v 2 ) ​ ( w − s ) = 2 ​ η ​ w ​ ( s − w ) \dot{w}=\dot{u}\,v+u\,\dot{v}=-\eta(u^{2}+v^{2})(w-s)=2\eta\,w(s-w) (using u 2 = v 2 = w u^{2}=v^{2}=w at balance): logistic, with solution w ⁡ ( t ) = s ​ ( 1 + ( s / w 0 − 1 ) ​ e − 2 ​ η ​ s ​ t ) − 1 w(t)=s\big(1+(s/w_{0}-1)\,e^{-2\eta st}\big)^{-1} . The head-fit condition w 1 ​ ( T 1 ) = ( 1 − δ ) ​ s 1 w_{1}(T_{1})=(1-\delta)s_{1} gives e 2 ​ η ​ s 1 ​ T 1 = 1 − δ δ ​ ( s 1 − w 0 ) / w 0 ≤ 1 − δ δ ​ s 1 / w 0 e^{2\eta s_{1}T_{1}}=\tfrac{1-\delta}{\delta}\,(s_{1}-w_{0})/w_{0}\leq\tfrac{1-\delta}{\delta}\,s_{1}/w_{0} . For the tail, the exact solution obeys w ⁡ ( t ) ≤ s ​ w 0 s − w 0 ​ e 2 ​ η ​ s ​ t ≤ 2 ​ w 0 ​ e 2 ​ η ​ s ​ t w(t)\leq\tfrac{s\,w_{0}}{s-w_{0}}\,e^{2\eta st}\leq 2w_{0}\,e^{2\eta st} for w 0 ≤ s / 2 w_{0}\leq s/2 , so w 2 ​ ( T 1 ) ≤ 2 ​ w 0 ​ ( e 2 ​ η ​ s 1 ​ T 1 ) ρ ≤ 2 ​ ( 1 − δ δ ) ρ ​ s 1 ρ ​ w 0 1 − ρ = C δ , ρ ​ s 1 ​ ( w 0 s 1 ) 1 − ρ . w_{2}(T_{1})\;\leq\;2w_{0}\big(e^{2\eta s_{1}T_{1}}\big)^{\rho}\;\leq\;2\Big(\tfrac{1-\delta}{\delta}\Big)^{\rho}s_{1}^{\,\rho}\,w_{0}^{\,1-\rho}\;=\;C_{\delta,\rho}\,s_{1}\Big(\tfrac{w_{0}}{s_{1}}\Big)^{1-\rho}.

(ii) The h ≡ η h\equiv\eta flow acts on every factor by replacing its update with η \eta times the matrix sign of its gradient. In the scalar case this gives u ˙ = − η ​ sign ​ ( v ⁡ ( w − s ) ) = η \dot{u}=-\eta\,\mathrm{sign}(v(w-s))=\eta , so that u u grows linearly in time, and likewise v ˙ = η \dot{v}=\eta , for 0 < w < s 0<w<s . Hence each of the two factors increases like w 0 + η ​ t \sqrt{w_{0}}+\eta t , i.e. w = w 0 + η ​ t \sqrt{w}=\sqrt{w_{0}}+\eta t until w w reaches the threshold s s ; there the gradient becomes zero, and since sign ⁡ ( 0 ) = 0 \mathrm{sign}(0)=0 (as also follows from msign ⁡ ( 0 ) = 0 \mathrm{msign}(0)=0 by Proposition 4.2 (3)), the mode stops. So the capped dynamics are the stopped flow. Mode i i therefore fits at t i = ( s i − w 0 ) / η t_{i}=(\sqrt{s_{i}}-\sqrt{w_{0}})/\eta , and the head reaches ( 1 − δ ) ​ s 1 (1-\delta)s_{1} at T 1 = ( ( 1 − δ ) ​ s 1 − w 0 ) / η T_{1}=\big(\sqrt{(1-\delta)s_{1}}-\sqrt{w_{0}}\big)/\eta . For δ < 1 − s 2 / s 1 \delta<1-s_{2}/s_{1} we have ( 1 − δ ) ​ s 1 > s 2 (1-\delta)s_{1}>s_{2} , hence t 2 < T 1 t_{2}<T_{1} and w 2 ​ ( T 1 ) = s 2 w_{2}(T_{1})=s_{2} exactly, with no dependence on w 0 w_{0} . ∎

## Appendix C Experimental details

##### Sensing task (§ 5 –§ 7 , § 8 ).

• Target and measurements. X ∗ = U ∗ ​ V ∗ ⁣ ⊤ X^{\ast}=U^{\ast}V^{\ast\top} , U ∗ , V ∗ ∈ ℝ 40 × 3 U^{\ast},V^{\ast}\in\mathbb{R}^{40\times 3} i.i.d. Gaussian scaled by 1 / 3 1/\sqrt{3} ; m = 462 = 2 ​ dof m=462=2\,\mathrm{dof} , dof = 3 ​ ( 80 − 3 ) = 231 \mathrm{dof}=3(80-3)=231 ; A i A_{i} i.i.d. standard Gaussian; loss = mean i ​ ( ⟨ A i , W ⟩ − y i ) 2 =\mathrm{mean}_{i}(\left\langle A_{i},W\right\rangle-y_{i})^{2} .

• Factors. U , V ∈ ℝ 40 × 40 U,V\in\mathbb{R}^{40\times 40} initialized 𝒩 ⁡ ( 0 , 10 − 6 ) \mathcal{N}(0,10^{-6}) entrywise ( 10 − 3 10^{-3} scale).

• Budget. 2 × 10 4 2\times 10^{4} steps (extended where noted); interpolation bar 10 − 7 10^{-7} (checked every 200 steps); 3 paired seeds {42, 123, 456}.

Optimizer hyperparameters: • Adam, scalar-Adam. β = ( 0.9 , 0.999 ) \beta=(0.9,0.999) , ϵ = 10 − 8 \epsilon=10^{-8} .

• RMSProp. Classical form, second-moment decay 0.99 0.99 and no first moment, ϵ = 10 − 8 \epsilon=10^{-8} .

• Muon. Momentum 0.9, Newton–Schulz 5 iterations, coefficients (3.4445, − 4.7750 -4.7750 , 2.0315) ( Jordan et al., 2024 ) .

• Shampoo. Accumulated (non-EMA) L , R L,R , damping λ = 1 \lambda=1 , inverse-root refresh every 20 steps.

• Adafactor. Row/column EMA factored second moment.

• Lion. β 1 = 0.9 \beta_{1}=0.9 inner, 0.99 EMA.

• signum. Sign of an EMA momentum with β 1 = 0.9 \beta_{1}=0.9 (Adam’s β 1 \beta_{1} ) and no second moment, no clipping.

Learning-rate grids per method span ≥ \geq one decade around the interpolating range (curves for five reference methods in Appendix D ). The dial’s fixed-step arm (§ 7 ) re-runs the full p p -sweep at each single rate of the shared grid { 10 − 3 , 3 × 10 − 3 , 10 − 2 , 3 × 10 − 2 , 10 − 1 } \{10^{-3},3\times 10^{-3},10^{-2},3\times 10^{-2},10^{-1}\} , same update rule, seeds, and interpolation bar ( experiments/precond_dial_fixed_lr.py ).

Both dial arms use a 3 × 10 4 3\times 10^{4} -step budget ( experiments/precond_dial_scalar_check.py ), longer than the zoo’s 2 × 10 4 2\times 10^{4} . That CPU implementation follows Algorithm 1 line by line; two other code paths place ϵ \epsilon differently. The GPU port behind the n = 128 n{=}128 dial rows ( experiments/nibi/common.py ) omits the outer ϵ \epsilon for 0 < p < 1 0<p<1 and divides by s ¯ + ϵ \bar{s}+\epsilon rather than s ¯ + 2 ​ ϵ \bar{s}+2\epsilon at p = 0 p{=}0 . The released flowadam package, used for the FlowAdam- p p rows, folds the bias correction into the step size at p = 1 p{=}1 and adds ϵ \epsilon to v \sqrt{v} before that factor—TensorFlow’s Adam convention rather than Algorithm 1 ’s—for an effective ϵ / 1 − β 2 t \epsilon/\sqrt{1-\beta_{2}^{\,t}} . At ϵ = 10 − 8 \epsilon=10^{-8} neither departure reaches the digits we report. Its shared scalar for p < 1 p<1 is the geometric mean rather than Algorithm 1 ’s RMS, so the FlowAdam- p p rows sit on the geometric-mean dial of § 7 ’s footnote; § 10 reads their gain against the RMS dial’s 0.201 0.201 rather than the geometric-mean endpoint 0.229 0.229 , which is the conservative of the two.

##### Attention task (§ 6 ).

• Task. Modular addition modulo 47 on input sequences of the form ( a , b , = ) (a,b,{=}) , with 40% of the data used for training (fixed split).

• Model. A 2-layer transformer with 4 attention heads, model dimension d model = 64 d_{\mathrm{model}}=64 , using RMSNorm ( Zhang & Sennrich, 2019 ) in pre-normalization configuration and SiLU-activated ( Elfwing et al., 2018 ) MLPs of width 256. The model omits biases and does not apply query–key normalization (qk-norm), so the gauge holds exactly.

• Training. Full-batch optimization over 1500 steps, executed deterministically on CPU ( torch.use_deterministic_algorithms ).

• Gauge. An orthogonal transformation A h A_{h} per head, derived via QR decomposition of a Gaussian matrix, applied to the row-blocks of W Q W_{Q} and W K W_{K} .

• Learning rates. Adam uses 10 − 3 10^{-3} ; SGD with heavy-ball momentum ( 0.9 0.9 , unnormalized, as for Muon’s momentum here) uses 0.5 0.5 ; scalar-Adam is set to 3 × 10 − 3 3\times 10^{-3} , while Muon uses 0.02 0.02 on matrix parameters, with the standard hybrid applying Adam at 10 − 3 10^{-3} to the embeddings and the output head.

Drift is defined as the relative ℓ 2 \ell_{2} logit distance on the validation set, ‖ Z − Z ′ ‖ F / 1 2 ​ ( ‖ Z ‖ F + ‖ Z ′ ‖ F ) \left\lVert Z-Z^{\prime}\right\rVert_{F}\big/\tfrac{1}{2}(\left\lVert Z\right\rVert_{F}+\left\lVert Z^{\prime}\right\rVert_{F}) , where Z Z and Z ′ Z^{\prime} are the validation logit matrices of the two model copies. QK drift is computed the same way, but on W Q ⊤ ​ W K W_{Q}^{\top}W_{K} for each attention head and then aggregated across heads. The symmetrized ratio is used in both cases.

##### Hyperspectral task (§ 9 ).

• Scenes. The Indian Pines dataset is the corrected AVIRIS cube ( Baumgardner et al., 2015 ) , with 2000 randomly sampled pixels over 200 spectral bands (water-absorption bands removed), and the Pavia University dataset is 2000 pixels in 103 bands from the ROSIS sensor. Both scenes are taken from Graña et al. (2021) with band-wise mean removed.

• Model and loss. The factorization rank is 48, at initialization scale 10 − 2 10^{-2} ; no weight decay (wd = 0 =0 ); entrywise completion loss minimized over the entries in the observed mask.

• Densities. Two densities per scene were selected, for which m / dof 24 ≈ 1.15 m/\mathrm{dof}_{24}\approx 1.15 and 1.9 1.9 (corresponding to 0.15 / 0.25 0.15/0.25 for Indian Pines and 0.28 / 0.46 0.28/0.46 for Pavia).

• Protocol. Checkpoints are selected by matching training-set loss values in the range 3 × 10 − 3 3{\times}10^{-3} to 10 − 5 10^{-5} ; learning rates are chosen by the train-only selection rule of Appendix D.6 . The experiments were repeated with four random seeds (two for the learning-rate selection and two independent runs), with results aggregated across all four. Since the selection procedure only uses the training set, it carries no information about the test set.

The degrees of freedom are counted at rank 24 24 , which is where each scene’s singular spectrum begins to stabilize (leaving a residual energy of about 0.10 0.10 for Indian Pines). Hence the ratio m / dof 24 m/\mathrm{dof}_{24} characterizes how underdetermined the underlying signal is. The model is over-parameterized to rank 48 48 , so that the implicit bias, rather than a rank constraint, selects the solution. Every reading in § 9 and Table 5 —both scenes, both densities—comes from the same GPU float64 harness under the same train-only rule; the 3 × 10 − 5 3\times 10^{-5} and ≤ 10 − 5 \leq 10^{-5} figures are that harness read at two depths of fit.

##### FlowAdam experiments (§ 10 ).

Following Singh & Sheel (2026) , we test two clipping modes, percoord (each entry clipped to [ − 1 , 1 ] [-1,1] ) and globalnorm (the whole velocity rescaled to norm ≤ c \leq c , so c c is the threshold referred to in § 10 ), and a varying precond_power , on a step budget of 3 × 10 4 3\times 10^{4} ( experiments/flowadam_upgrade.py ), longer than the zoo’s 2 × 10 4 2\times 10^{4} ; for reference, the early-stopping audit of Appendix D.8 goes up to 1.2 × 10 5 1.2\times 10^{5} . We use the same three paired seeds and the same 10 − 7 10^{-7} interpolation threshold. Two configurations failed to meet that threshold within the allotted steps: FlowAdam- p = 0 p{=}0 got to 7 × 10 − 6 7\times 10^{-6} , and global-norm clipping at p = 1 p{=}1 reached 6 × 10 − 7 6\times 10^{-7} . Appendix D.8 continues the former to a true 10 − 7 10^{-7} and finds the same 0.1691 0.1691 ; the latter’s 0.220 0.220 is a deep fit rather than an interpolating reading, and is treated as such in comparisons. The evaluations were carried out in the dense, well-conditioned regime of § 10.1 , tuned versus tuned, with the same same-placement L2 control (identical L2 term in the loss for the Adam baseline) across five paired seeds.

##### Comparing values across tables.

Some results are repeated with minor variations. The differences come from different learning-rate grids (finer ones in the diagnostic experiments of Appendix D.3 ), different numbers of seeds (always at least three), and different step limits—but not from a difference in outcome, and overall the finer grids produce slightly lower values. Two observations stand out. First, the zoo’s selection grids stop above the smallest interpolating learning rate, whereas the diagnostic grids of Appendix D.3 go below it, so a flow-limit reading can fall under the corresponding Table 2 row (Remark A.5 ): GD 0.113 0.113 versus 0.131 0.131 , scalar-Adam 0.165 0.165 versus 0.201 0.201 , and Adam fixed near 0.56 0.56 – 0.58 0.58 .

Second, because the dial evaluates p p on a single shared grid at a coarser resolution, its endpoint values reflect the rates that grid selects rather than those chosen by the zoo or the ladder. For instance, at p = 1 p{=}1 the value reads 0.570 0.570 compared to the zoo’s 0.5734 0.5734 , and at p = 0 p{=}0 with n = 128 n{=}128 it reads 0.40 0.40 compared to the ladder’s 0.345 0.345 . Monotonicity in p p remains consistent regardless. Discrepancies smaller than 0.01 0.01 across tables stem from grid effects, and we do not highlight them individually.

## Appendix D The control battery

All key results reported in the paper were re-evaluated under the following control conditions. None of them showed meaningful changes beyond noise. The only estimate that became more precise with additional random seeds—the phase boundary τ ∗ \tau^{\ast} —is noted in Appendix D.1 . Two controls actually strengthened the conclusions rather than simply verifying them: the can/cannot framing detailed in Appendix D.2 , and the observation regarding RMSProp’s conservative behavior.

### D.1 Schedule symmetrization

Constant-update-norm methods (Muon, signum, Lion) require decay to interpolate; the concern is that decay itself (small late steps) shifts the bias. We ran every method under three schedules (Table 6 ): constant (early-stopped at interpolation), cosine (early-stopped), and cosine over the full 2 × 10 4 2\times 10^{4} -step horizon with no early stopping, the exact treatment Muon received, including its small late steps.

† train-loss floor 6 × 10 − 2 6\times 10^{-2} : Muon does not interpolate at constant lr, which is the reason decay is required. Shown for completeness.

The phase diagram of § 8 was also redone with an added cosine row for each method. Recovery is virtually identical across all τ \tau for GD, Adam, and Shampoo, to within the noise, and Muon is shifted by no more than 0.02 0.02 . The relative ordering of the methods is the same under both schedules, constant and decaying, with Adam having the worst performance across all six τ \tau in both cases. Increasing the number of random seeds from 3 3 to 10 10 brought the estimated crossover point from τ ∗ ≈ 0.35 \tau^{\ast}\approx 0.35 to τ ∗ ≈ 0.2 \tau^{\ast}\approx 0.2 ( ≈ 4 % \approx 4\% tail energy), but the method ranking and the schedule-invariance of that ranking still hold.

A separate sweep at n = 128 n{=}128 reproduces the GD < < Shampoo < < Adam ordering in the low- τ \tau part of the grid. However, we prefer to report this experiment only qualitatively, because on the ladder’s own per-size grid and at τ = 0 \tau{=}0 , untuned Shampoo instead sits above Adam at n = 128 n{=}128 (Table 11 ). This is not a contradiction to our main result, since the per-size grid and the procedure are different. Within this sweep, the same untuned damping shows again at higher τ \tau : starting from τ = 0.25 \tau{=}0.25 , Shampoo edges above Adam, by less than 0.01 0.01 throughout and well within the seed spread of either method, so unlike at n = 40 n{=}40 , Adam is not universally the worst optimizer at n = 128 n{=}128 . This sweep used a different two-rate grid from the one used in the main ladder experiments (Appendix D.9 ). On that grid Muon does not recover cleanly at n = 128 n{=}128 , its value being 0.54 0.54 at τ = 0 \tau{=}0 instead of 0 0 , so its crossing is not added to the table. It should be noted, nevertheless, that the ladder’s broader grid does intersect Muon’s interpolation band at n = 128 n{=}128 , where Muon indeed recovers fully (Table 11 ), so the difference is in the coverage, not in the result.

### D.2 The learning-rate axis

Choosing a method’s learning rate based on optimal recovery constitutes selection by outcome, while selecting based on fastest interpolation ignores bias considerations—yet leads each method to its most bias-distorted step size (even Muon degrades to 0.91 0.91 recovery at its fastest-interpolating rate). Neither criterion on its own supports a conclusion about implicit bias. We therefore evaluate recovery as a function of learning rate (Appendix Table 7 ) and discuss an achievability asymmetry: across the tested grids, every equivariant method reaches good recovery at some learning rate, whereas no coordinate-wise approach does at any rate, even its best (Adam’s recovery stays confined to 0.56 0.56 – 0.58 0.58 over two orders of magnitude).