Sharpness-Aware Minimization for Efficiently Improving Generalization — part 4/4
Cleaned-text code points [47257, 51022). Bibliography omitted here; source.md includes it.
Our theorem is derived for p = 2 p=2 , although generalizations can be considered for p ∈ [ 1 , + ∞ ] p\in[1,+\infty] (the expression of the bound becoming way more involved). Empirically, we validate that the choice p = 2 p=2 is optimal by training a wide ResNet on CIFAR-10 with SAM for p = ∞ p=\infty (in which case we have ϵ ^ ( 𝒘 ) = ρ sign ( ∇ 𝒘 L 𝒮 ( 𝒘 ) ) \hat{\bm{\epsilon}}(\bm{w})=\rho\sign{(\nabla_{\bm{w}}L_{\mathcal{S}}(\bm{w}))} ) and p = 2 p=2 (giving ϵ ^ ( 𝒘 ) = ρ ‖ ∇ 𝒘 L 𝒮 ( 𝒘 ) ‖ 2 2 ( ∇ 𝒘 L 𝒮 ( 𝒘 ) ) \hat{\bm{\epsilon}}(\bm{w})=\frac{\rho}{||\nabla_{\bm{w}}L_{\mathcal{S}}(\bm{w})||_{2}^{2}}(\nabla_{\bm{w}}L_{\mathcal{S}}(\bm{w})) ). We do not consider the case p = 1 p=1 which would give us a perturbation on a single weight. As an additional ablation study, we also use random weight perturbations of a fixed Euclidean norm: ϵ ^ ( 𝒘 ) = ρ ‖ 𝒛 ‖ 2 2 𝒛 \hat{\bm{\epsilon}}(\bm{w})=\frac{\rho}{||\bm{z}||_{2}^{2}}\bm{z} with 𝒛 ∼ 𝒩 ( 𝟎 , 𝑰 d ) \bm{z}\sim{\mathcal{N}}(\bm{0},\bm{I}_{d}) . We report the test accuracy of the model in figure 6 .
We observe that adversarial perturbations outperform random perturbations, and that using p = 2 p=2 yield superior accuracy on this example.
### C.6 Several Iterations in the Inner Maximization
To empirically verify that the linearization of the inner problem is sensible, we trained a WideResNet on the CIFAR datasets using a variant of SAM that performs several iterations of projected gradient ascent to estimate max ϵ L ( 𝒘 + ϵ ) \max_{\bm{\epsilon}}L(\bm{w}+\bm{\epsilon}) . We report the evolution of max ϵ L ( 𝒘 + ϵ ) − L ( 𝒘 ) \max_{\bm{\epsilon}}L(\bm{w}+\bm{\epsilon})-L(\bm{w}) during training (where L L stands for the training error rate computed on the current batch) in Figure 7 , along with the test accuracy and the estimated sharpness ( max ϵ L ( 𝒘 + ϵ ) − L ( 𝒘 ) \max_{\bm{\epsilon}}L(\bm{w}+\bm{\epsilon})-L(\bm{w}) ) at the end of training in Table 11 ; we report means and standard deviations across 20 runs.
For most of the training, one projected gradient step (as used in standard SAM) is sufficient to obtain a good approximation of the ϵ \bm{\epsilon} found with multiple inner maximization steps. We however observe that this approximation becomes weaker near convergence, where doing several iterations of projected gradient ascent yields a better ϵ \bm{\epsilon} (for example, on CIFAR-10, the maximum loss found on each batch is about 3% more when doing 5 steps of inner maximization, compared to when doing a single step). That said, as seen in Table 11 , the test accuracy is not strongly affected by the number of inner maximization iterations, though on CIFAR-100 it does seem that several steps outperform a single step in a statistically significant way.
## Instructions for reporting errors
We are continuing to improve HTML versions of papers, and your feedback helps enhance accessibility and mobile support. To report errors in the HTML that will help us improve conversion and rendering, choose any of the methods listed below:
Click the "Report Issue" ( ) button, located in the page header.
Tip: You can select the relevant text first, to include it in your report.
Our team has already identified the following issues . We appreciate your time reviewing and reporting rendering errors we may not have found yet. Your efforts will help us improve the HTML versions for all readers, because disability should not be a barrier to accessing research. Thank you for your continued support in championing open access for all.
Have a free development cycle? Help support accessibility at arXiv! Our collaborators at LaTeXML maintain a list of packages that need conversion , and welcome developer contributions .