The Loss Does Not See the Basis, but Adam Does

arxiv:2608.05136

Original source · Paper JSON

Data, training, optimization · 2026-08-05

Cites (1) · Cited by (0)

Complete cleaned Markdown · Source Markdown including bibliography

Read in parts

1. Part 1
2. Part 2
3. Part 3
4. Part 4
5. Part 5
6. Part 6
7. Part 7
8. Part 8
9. Part 9
10. Part 10
11. Part 11
12. Part 12

Contents

• The Loss Does Not See the Basis, but Adam Does
• Abstract
• 1 Introduction
• Contributions.
• Scope.
• 2 Related work
• Implicit bias of gradient methods on factored models.
• Implicit bias of Adam and adaptive methods.
• Optimizer design, symmetry, and structured methods.
• Our own prior work.
• 3 Setup: gauge, task, and protocol
• Factored model and gauge transformation.
• Definition 3.1 (Gauge equivariance of an optimizer) .
• Lemma 3.2 (Orthogonal gauge as maximal isometric symmetry) .
• Task: recovery in the interpolation regime.
• Protocol.
• 4 Theory: equivariance as a classifier of update rules
• Lemma 4.1 (Gradient covariance) .
• Proposition 4.2 (Equivariant class) .
• Proposition 4.3 (Nonlinear coordinate-wise maps break the gauge) .
• Proposition 4.4 (Adam has an exact first-step gauge defect) .
• Theorem 4.5 (Structure: equivariant memoryless rules are Gram-determined left preconditioners) .
• Theorem 4.6 (Transfer theorem) .
• Remark 4.7 (Scope of the transfer, and what equivariance does not buy) .
• 5 The optimizer zoo
• Design.
• Result.
• Controls (Appendix D ).
• 6 The same gauge lives in attention heads
• Attention has the same symmetry.
• Twin protocol.
• Results (Table 3 , Figure 2 ).
• Scale, and real stochastic training.
• A consequence for model merging.
• 7 Isolating preconditioner anisotropy with a dial
• 8 Inside the equivariant class: the spectral-tail phase diagram
• Design.
• The two axes in the sweep.
• An analytically tractable model.
• Proposition 8.1 (Greedy vs. equal-rate schedules: a solvable two-timescale scenario) .
• 9 Real data at matched training loss
• Validation and a second benchmark.
• 10 Repairing our own optimizer using the criterion
• 10.1 Where the bias succeeds and where it does not
• 11 Discussion and limitations
• Scale.
• Two exceptions we left untuned.
• Not a prescription for practice.
• Open theoretical directions.
• Stochasticity.
• Further consequences.
• Reproducibility.
• Acknowledgments
• Use of Large Language Models
• Appendix A Inside the equivariant class
• Proposition A.1 (The spectral transfer function: the within-class coordinate) .
• Remark A.2 (Stateful rules and right preconditioners) .
• Corollary A.3 (Deeper factorizations) .
• Proposition A.4 (Balancedness: a special property of GD-type flows, not a general invariance) .
• Remark A.5 (Comments on discretization and step-size choice) .
• Appendix B Proofs
• B.1 Lemma 3.2 (maximal isometric gauge)
• Proof.
• B.2 Lemma 4.1 (gradient covariance)
• Proof.
• B.3 Proposition 4.2 (equivariant class)
• Proof.
• B.4 Proposition 4.3 (coordinate-wise rules break the gauge)
• Proof.
• B.5 Proposition 4.4 (exact first-step Adam defect)
• Proof.
• B.6 Theorem 4.5 (structure)
• Proof.
• B.7 Proposition A.1 (spectral transfer)
• Proof.
• B.8 Corollary A.3 (deeper factorizations)
• Proof.
• B.9 Theorem 4.6 (transfer theorem)
• Proof.
• B.10 Proposition A.4 (balancedness)
• Proof.
• B.11 Proposition 8.1 (greedy vs. equal-rate phase boundary)
• Proof.
• Appendix C Experimental details
• Sensing task (§ 5 –§ 7 , § 8 ).
• Attention task (§ 6 ).
• Hyperspectral task (§ 9 ).
• FlowAdam experiments (§ 10 ).
• Comparing values across tables.
• Appendix D The control battery
• D.1 Schedule symmetrization
• D.2 The learning-rate axis
• D.3 Learning-rate curves and selection rules
• D.4 Initialization scale
• D.5 Attention twins across seeds, initialization draws, and noise levels
• Scaling the twin diagnostic (GPU, float64 ).
• D.6 Hyperspectral learning-rate hygiene
• Two limits of this grid.
• D.7 Interpolation-bar audits
• D.8 FlowAdam-p is not an artifact of early stopping
• D.9 The H100 replication ladder
• Discussion around the ladder.
• D.10 Recovery without equivariance: annealed sign descent at long horizons
• Appendix E Extended related work
• Transformation invariance in optimizer design.
• Symmetry-driven accounts of implicit bias.
• Muon and structured optimizers.
• Appendix F Relation to invariant-optimizer design
• Instructions for reporting errors