Sharpness-Aware Minimization for Efficiently Improving Generalization

arxiv:2010.01412

Original source · Paper JSON

Data, training, optimization · 2020-10-03

Cites (0) · Cited by (4)

Complete cleaned Markdown · Source Markdown including bibliography

Read in parts

1. Part 1
2. Part 2
3. Part 3
4. Part 4

Contents

• Sharpness-Aware Minimization for Efficiently Improving Generalization
• Abstract
• 1 Introduction
• 2 Sharpness-Aware Minimization (SAM)
• Theorem (stated informally) 1 .
• 3 Empirical Evaluation
• 3.1 Image Classification From Scratch
• 3.2 Finetuning
• 3.3 Robustness to Label Noise
• 4 Sharpness and Generalization Through the Lens of SAM
• 4.1 m m -sharpness
• 4.2 Hessian Spectra
• 5 Related Work
• 6 Discussion and Future Work
• 7 Acknowledgments
• Appendix A Appendix
• A.1 PAC Bayesian Generalization Bound
• Theorem 2 .
• Proof.
• Appendix B Additional Experimental Results
• B.1 SVHN and Fashion-MNIST
• Appendix C Experiment Details
• C.1 Hyperparameters for Experiments
• C.2 Finetuning Details
• C.3 Experimental results with ρ = 0.05 \rho=0.05
• C.4 Ablation of the Second Order Terms
• C.5 Choice of p-norm
• C.6 Several Iterations in the Inner Maximization
• Instructions for reporting errors