NorMuon: Making Muon more efficient and scalable

arxiv:2510.05491

Original source · Paper JSON

Data, training, optimization · 2025-10-07

Cites (3) · Cited by (0)

Complete cleaned Markdown · Source Markdown including bibliography

Read in parts

1. Part 1
2. Part 2
3. Part 3

Contents

• NorMuon: Making Muon more efficient and scalable
• Abstract
• 1 Introduction
• 2 Related Works and Background
• 2.1 Related Works
• 2.2 Background: Muon optimizer
• 3 Method
• 3.1 NorMuon
• 3.2 Distributed NorMuon
• 3.3 Overhead Analysis
• 4 Experiments
• 4.1 Experiments on 1.1B and 5.4B Models
• 4.1.1 Setup
• 4.1.2 Main results
• 4.1.3 Ablation studies.
• 4.1.4 Computational and Memory Overhead Analysis
• 4.2 Experiments on Modded-NanoGPT
• 5 Conclusion
• Acknowledgments
• Appendix A Additional Experiments
• A.1 Adam-mini’s results on optimization geometry analysis
• A.2 Ablation Experiments on Modded-NanoGPT
• Appendix B Implementation Details
• Appendix C Experiment Setup of Modded-NanoGPT
• Instructions for reporting errors