The Loss Does Not See the Basis, but Adam Does — part 6/12

Cleaned-text code points [76576, 91756). Bibliography omitted here; source.md includes it.

A second independent hyperspectral dataset, Pavia University, matched to the same two density levels, exhibits a consistent trend: + 43.8 % +43.8\% improvement at m / dof ≈ 1.15 m/\mathrm{dof}\approx 1.15 and + 22.9 % +22.9\% at m / dof ≈ 1.9 m/\mathrm{dof}\approx 1.9 (results consistent across all 4 4 seeds; see Table 5 ). Muon does not outperform GD on either dataset or density, and its effective rank stays near the model limit ( 46 46 – 47 47 of a capacity of 48 48 ) for most of the matched-loss path, declining only in its final levels, earlier on Pavia than on Indian Pines—the tail regime of § 8 . On Indian Pines, Muon is actually worse than Adam, the opposite effect to the one observed in the synthetic experiments (Table 4 ), where Adam was significantly worse at every τ \tau ; on Pavia, where that collapse does run to completion (rank 10 10 and 8 8 of 48 48 ), Muon lands back ahead of Adam at both densities. While being equivariant influences which mechanisms a method can access, it does not guarantee improvements in performance over the alternatives in all spectral settings. Muon’s persistently high effective rank suggests that it is the schedule dimension that determines the behavior of a method here, not the gauge. Scalar-Adam is absent from the table because its rate grid here was undersampled and it does not reproduce the phenomenon (Appendix D.6 ). The GD–Adam gap carries the main matched-loss conclusion of this section (Table 5 ).

## 10 Repairing our own optimizer using the criterion

FlowAdam ( Singh & Sheel, 2026 ) introduces clipped gradient-flow velocity into Adam’s momentum mechanism, aiming to incorporate the dynamics responsible for bias as described in the memoryless, idealized form of Theorem 4.6 —though the actual stateful hybrid version falls outside the theorem’s formal reach. Without weight decay, however, it only partially closes the performance gap to gradient descent (GD), reducing the recovery error from 0.573 0.573 to 0.347 0.347 , compared to GD’s 0.131 0.131 . The symmetry criterion exposes the flaw in the design of this algorithm: coordinate-wise velocity clipping, v ↦ clip ⁡ ( v , − c , c ) v\mapsto\mathrm{clip}(v,-c,c) , acts independently on each coordinate and so breaks gauge symmetry, by Proposition 4.3 . In short, the component designed to restore gauge-consistent behavior breaks it by the very act of being there.

By design, the clipping is the binding constraint: as its threshold c → ∞ c\to\infty the clip switches off and the modified dynamics reduce to the underlying gradient flow. The appropriate way to modify the clipping is to restrict the whole velocity vector’s norm, not individual coordinates, which constrains magnitude without altering direction. It is this correction that brings FlowAdam’s bias to match the level of GD in the continuum limit (as c → ∞ c\to\infty it recovers at 0.113 0.113 , which is GD’s own value at its smallest interpolating rate in Table 7 , versus the 0.131 0.131 of GD’s benchmark row). Applied alone, it improves the recovery error by 37 % 37\% ( 0.347 → 0.220 0.347\to 0.220 , this run stopping at a training loss of 6 × 10 − 7 6\times 10^{-7} rather than the 10 − 7 10^{-7} threshold, so it is a deep fit and not an interpolating reading; Appendix C ). The clip is all that this repairs, though: the surrounding Adam step still preconditions per coordinate, so the method as a whole remains gauge-breaking, and 0.220 0.220 is where that leaves it. Combined with the preconditioner-anisotropy dial from § 7 (making it FlowAdam- p p ), it gives the best recovery among the Adam-type variants we test ( 0.169 0.169 versus Adam’s 0.573 0.573 ), with the flow contributing a further + 15.9 % +15.9\% over the dial alone ( 0.169 0.169 vs. 0.201 0.201 ) at the same learning rate. Furthermore, its effective rank ( 4.8 4.8 – 5.4 5.4 across different random seeds) is the closest among all the considered Adam-type algorithms to GD’s 4.51 4.51 .

The + 15.9 % +15.9\% improvement is achieved at the same learning rate (both at 10 − 3 10^{-3} , as per the dial’s anchoring procedure in § 7 ), and should not be viewed as an improvement in the asymptotic regime: on finer grids with smaller learning rates the two methods converge empirically, the dial alone reaching 0.143 0.143 ( n = 40 n{=}40 ) / 0.40 0.40 ( n = 128 n{=}128 ) while FlowAdam- p p achieves nearly identical results at 0.148 0.148 / 0.42 0.42 . That n = 40 n{=}40 cell is the slowest to fit and clears the interpolation bar only past the sweep’s 3 × 10 4 3\times 10^{4} steps; continued to a true 10 − 7 10^{-7} , all ten seeds cross by 6 × 10 4 6\times 10^{4} and read the same 0.1475 ± 0.0357 0.1475\pm 0.0357 . The evidence therefore points to a consistent improvement at fixed learning rates rather than a distinct limiting solution, which is what keeps the anchor-protocol comparison central. The improvement is not due to early stopping: extending training to 1.2 × 10 5 1.2\times 10^{5} steps shows stable recovery values from a training loss of 10 − 4 10^{-4} down to the stringent 10 − 7 10^{-7} interpolation threshold (mean 0.1694 → 0.1691 0.1694\to 0.1691 , with individual seeds shifting by no more than 3 × 10 − 4 3\times 10^{-4} ; Appendix D.8 ). This trajectory—from coordinate-wise clipping to an unexplained performance ceiling, then to a global-norm correction—mirrors, on a smaller scale, the broader validation of the paper’s symmetry criterion.

### 10.1 Where the bias succeeds and where it does not

Implicit bias lacks a separate, adjustable strength parameter. Across the three dimensions explored in the FlowAdam setup of § 10 (varying observation density, signal conditioning, and explicit regularization, as per Appendix C ), a certain pattern emerges regarding the advantage of flow-based or equivariant bias over a well-chosen explicit weight decay: the former demonstrates clear benefits only when the optimal amount of regularization is relatively low and structured. When the signal is dense and well-conditioned, the method consistently outperforms its Adam equivalent at the same L2 placement, achieving gains of around 3 3 – 6 % 6\% for all five random seeds in our experiments (results not tabulated). On the other hand, in the more difficult regime, where strong regularization is necessary (e.g. with sparse observations or ill-conditioned, low-rank targets), using the flow hurts performance compared to training with Adam directly.

Three more limitations stem from the theoretical properties of the described procedure rather than from empirical observation. First, phenomena such as grokking ( Power et al., 2022 ) likely require persistent forces that have no analogue in the geometric flow, such as weight decay, to bias the trajectory after task gradients have vanished. Second, in physics-informed neural networks (PINNs) ( Raissi et al., 2019 ) , stiff problems have gradients whose magnitudes differ vastly across coordinates ( Wang et al., 2021 ) . There, anisotropic (Adam-style) optimization is more beneficial than isotropic (SGD-style) optimization, as recently advocated in ℓ ∞ \ell_{\infty} geometry ( Xie & Li, 2024 ; Gronich & Vardi, 2026 ) . Third, LoRA fine-tuning ( Hu et al., 2022 ) factors the adapter weight matrix into lower-rank matrices, i.e. it introduces a rank constraint. By fixing the rank rather than learning it, LoRA breaks the mechanism discussed in this paper; it may, however, still find better solutions inside the lower-dimensional subspace if they exist.

## 11 Discussion and limitations

##### Scale.

The original experiments were deliberately modest in scope, using deterministic twins, matrices of modest size ( 40 × 40 40\times 40 to 2000 × 200 2000\times 200 ) and transformers of up to six layers, since establishing the mechanism required focusing on determinism and interpolation discipline rather than on raw size. To ground these claims, we scaled up a replication experiment on H100 GPUs, varying the problem size from n = 64 n{=}64 to n = 256 n{=}256 and using ten different random seeds (Appendix D.9 ). We tested attention twins with four and six layers and one character language model (Appendix D.5 ), and applied the method to a second hyperspectral dataset (§ 9 ). On the sensing ladder, gradient descent, scalar-Adam, and Muon significantly outperformed the coordinate-wise Adam-family methods at every size, with Muon exact up to n = 256 n{=}256 . On the hyperspectral scenes only the GD–Adam ordering survives: Muon loses to Adam on Indian Pines at both densities, and scalar-Adam’s grid there is undersampled (§ 9 , Appendix D.6 ). The attention gauge-split was generally scalable and robust to realistic stochastic training, and similar dynamics were observed in the second dataset.

##### Two exceptions we left untuned.

The distinctive two-cluster structure in Figure 1 is qualified by these two exceptions. Shampoo with damping ( λ = 1 \lambda{=}1 ) is evaluated at scale without parameter tuning, which means its absolute recovery degrades with increasing n n : it sits far below Adam at n = 40 n{=}40 , overtakes Adam at n = 64 n{=}64 , and remains above it thereafter, albeit not monotonically—it dips slightly below Adam again at n = 96 n{=}96 . The reason for this is that we kept all hyperparameters at their originally published values for all methods and problem sizes, varying only the learning rate. This means that Adam’s ( β 1 , β 2 , ϵ ) (\beta_{1},\beta_{2},\epsilon) , RMSProp’s decay hyperparameter, and Muon’s Newton–Schulz coefficients were kept as published. Tuning λ \lambda in Shampoo only at the sizes where it misbehaves would have amounted to outcome-based tuning, which § 3 advises against; hence we note its deterioration across scales instead. A similar consideration applies to the long-anneal variant of signum in Appendix D.10 , which drives a coordinate-wise method to zero recovery. In both cases the relative performance ordering of GD, scalar-Adam, and Muon remains the same, but taken together these two exceptions make the zoo’s empty gap a statement about this computational budget and this size range rather than about an asymptotic principle. Two further points: the n = 384 n{=}384 case probably has a better learning rate to be found, as Muon did not reach its interpolating rate on the grid we ran there; and several methods’ recovery values are budget-restricted estimates, with a decent amount of room for improvement from a wider hyperparameter search.

##### Not a prescription for practice.

With this, we think the results neither single out a particular method to be favored in practice nor suggest that GD is somehow better than Adam. In fair comparisons, with the same budget spent on tuning, no optimizer has been found to consistently outperform a well-tuned Adam ( Schmidt et al., 2021 ; Dahl et al., 2023 ) . It is important to note, finally, that what we compare is not how quickly a method optimizes the training objective, but which interpolant it selects at a fixed budget. Adam has also been shown to be more robust to issues such as gradient heterogeneity and heavy-tailed class imbalance ( Kunstner et al., 2024 ) , neither of which appears in this task.

##### Open theoretical directions.

Equivariance has been the geometric condition that allowed us to analyze the GD family, but the transfer theorem only applies to the common-scalar subclass (Remark 4.7 ). While equivariance alone does not specify the interpolant—ScaledGD is equivariant but equalizes the schedule (Remark 4.7 )—the non-equivariant annealed sign descent of Appendix D.10 shows that it is not strictly necessary either. Classifying interpolants thus requires consideration of the gauge and of the spectral schedule, and two directions seem natural for extending the theory as it stands. First, a full-dynamics theorem for the boundary in Table 4 (and Figure 4 ), along the lines of Kang et al. (2026) , would characterize the equivariant greedy-schedule regime in which the gradient-flow bias is maintained, with ScaledGD (schedule-equalizing) and signum (flattening, large-step) as its extremes along the two axes. Proposition 8.1 captures this in a decoupled two-factor model, leaving aside the interesting possibility of interacting schedules.

Second, a momentum-invariance lemma. Since the first-moment EMA is linear and respects the gauge, showing that it preserves the limit point of the flow would generalize the observed small-step trend for scalar-Adam into an exact equivalence with gradient flow, and would extend the transfer theorem to the momentum-augmented methods it currently sets aside.

##### Stochasticity.

Our twin models are both full-batch and deterministic. Mini-batch noise adds a diffusion term whose interaction with the gauge (and with Muon’s msign \mathrm{msign} -induced chaos) we have probed only empirically—in the character-language-model twins of § 6 and the noise-twin ladder of Appendix D.5 —but not characterized. The quotient-geometric framework of Aladrah et al. (2026) is a natural language for that interaction.

##### Further consequences.

The consequence for alignment-based model merging is in § 6 ; Appendix F takes up what the criterion implies for invariant-optimizer design.

##### Reproducibility.

The experiments seeking to understand the mechanism ran on CPU in minutes. Only the H100 replication ladder (Appendix D.9 ) and the bigger attention twins needed a GPU. The code, random seeds, and all control conditions (including schedule symmetrization, model selection criteria, learning-rate curves, initialization variations, and noise-twin scaling) are available at https://github.com/idevender/loss-basis-adam .

#### Acknowledgments

I thank Tarun Sheel for the collaboration on FlowAdam that motivated this line of work, and for his support. This research was enabled in part by computing resources provided by ACENET and the Digital Research Alliance of Canada (the Nibi cluster).

## Use of Large Language Models

LLMs were used for copy-editing only. All questions, proofs, experiments, and numbers are the author’s own and were checked against the derivations and the experiment logs.

## Appendix A Inside the equivariant class

Theorem 4.5 pins down the shape of an equivariant memoryless rule. This appendix collects what lives inside that shape: the spectral coordinate, the extension to stateful rules and to deeper factorizations, and what happens to balancedness.

###### Proposition A.1 (The spectral transfer function: the within-class coordinate) .

Suppose H = ψ ⁡ ( G ​ G ⊤ ) H=\psi(GG^{\top}) , where ψ \psi is a scalar-valued spectral function applied via functional calculus to the symmetric positive-semidefinite Gram matrix G ​ G ⊤ GG^{\top} —that is, ψ \psi acts as multiplication by ψ ⁡ ( λ ) \psi(\lambda) on each corresponding eigenspace. Given the singular value decomposition G = ∑ i σ i ​ u i ​ v i ⊤ G=\sum_{i}\sigma_{i}u_{i}v_{i}^{\top} , it follows that Φ ⁡ ( G ) = ∑ i h ⁡ ( σ i ) ​ u i ​ v i ⊤ , where h ⁡ ( σ ) ≔ ψ ⁡ ( σ 2 ) ​ σ . \Phi(G)\;=\;\sum_{i}h(\sigma_{i})\,u_{i}v_{i}^{\top},\qquad\text{where}\quad h(\sigma)\coloneqq\psi(\sigma^{2})\,\sigma.