The GAN is dead; long live the GAN! A Modern GAN Baseline — part 5/7

Cleaned-text code points [62595, 78559). Bibliography omitted here; source.md includes it.

Proof for the main theorem. Given previous lemmas, by choosing local coordinates θ ⁡ ( α , γ G ) \theta(\alpha,\gamma_{G}) and ψ ⁡ ( β , γ D ) \psi(\beta,\gamma_{D}) for ℳ G \mathcal{M}_{G} and ℳ D \mathcal{M}_{D} such that θ ∗ = 0 \theta^{*}=0 , ψ ∗ = 0 \psi^{*}=0 as well as ℳ G = 𝒯 θ ∗ ​ ℳ G = { 0 } k × ℝ n − k \displaystyle\mathcal{M}_{G}=\mathcal{T}_{\theta^{*}}\mathcal{M}_{G}=\{0\}^{k}\times\mathbb{R}^{n-k} (37) ℳ D = 𝒯 ψ ∗ ​ ℳ D = { 0 } l × ℝ m − l \displaystyle\mathcal{M}_{D}=\mathcal{T}_{\psi^{*}}\mathcal{M}_{D}=\{0\}^{l}\times\mathbb{R}^{m-l} (38) our proof is exactly the same as Mescheder et al . Mescheder et al. [2018] , Theorem 4.1.

## Appendix D Hyperparameters, training configurations, and compute

We implement our models on top of the official StyleGAN3 code base. While the loss function and the models are implemented from scratch, we reuse support code from the existing implementation whenever possible. This includes exponential moving average (EMA) of generator weights Karras et al. [2017] , non-leaky data augmentation Karras et al. [2020a] , and metric evaluation Karras et al. [2021] .

#### Training schedule.

To speed up the convergence early in training, we specify a cosine schedule for the following hyperparameters before they reach their target values: • Learning rate

• γ \gamma for R 1 R_{1} and R 2 R_{2} regularization

• Adam β 2 \beta_{2}

• EMA half-life

• Augmentation probability

We call this early training stage the burn-in phase. Burn-in length and schedule for each hyperparameter are listed in Table 19 for each experiment. A schedule for the EMA half-life can already be found in Karras et al . Karras et al. [2020a] , albeit they use a linear schedule. A lower initial Adam β 2 \beta_{2} is crucial to the initial large learning rate as it allows the optimizer to adapt to the gradient magnitude change much quicker. We use a large initial γ \gamma to account for that early in training: p θ p_{\theta} and p 𝒟 p_{\mathcal{D}} are far apart and a large γ \gamma smooths both distributions more aggressively which makes learning easier. Augmentation is not necessary until D D starts to overfit later on; thus, we set the initial augmentation probability to 0.

#### Dataset augmentation.

We apply horizontal flips and non-leaky augmentation Karras et al. [2020a] to all datasets where augmentation is enabled. Following Karras et al. [2020a] , we include pixel blitting, geometric transformations, and color transforms in the augmentation pipeline. We additionally include cutout augmentation which works particularly well with our model, although it does not seem to have much effect on StyleGAN2. We also find it beneficial to apply color transforms less often and thus set their probability multiplier to 0.5 while retaining the multiplier 1 for other types of augmentations. As previously mentioned, we apply a fixed cosine schedule to the augmentation probability rather than adjusting it adaptively as in Karras et al. [2020a] . We did not observe any performance degradation with this simplification.

#### Network capacity.

We keep the capacity distribution for each resolution the same as in Karras et al. [2020a] , Karras et al. [2021] . We place two residual blocks per resolution which makes our model roughly 3 × \times as deep, 1.5–3 × \times as wide as StyleGAN2 while maintaining the same model size on CIFAR-10 and FFHQ. For the ImageNet model, we double the number of channels which results in roughly 4 × \times as many parameters as the default StyleGAN2 configuration.

#### Mixed precision training.

We apply mixed precision training as in Karras et al. [2020a] , Karras et al. [2021] where all parameters are stored in FP32, but cast to lower precision along with the activation maps for the 4 highest resolutions. We notice that using FP16 as the low precision format cripples the training of our model. However, we see no problem when using BFloat16 instead.

#### Class conditioning.

For class conditional models, we follow the same conditioning scheme as in Karras et al. [2020a] . For G G , the conditional latent code z ′ z^{\prime} is the concatenation of z z and the embedding of the class label c c , specifically z ′ = concat ​ ( z , embed ​ ( c ) ) z^{\prime}=\text{concat}(z,\text{embed}(c)) . For D D , we use a projection discriminator Miyato & Koyama [2018] which evaluates the dot product of the class embedding and the feature vector D ′ ​ ( x ) D^{\prime}(x) produced by the last layer of D D , concretely D ⁡ ( x ) = embed ​ ( c ) ⋅ D ′ ​ ( x ) ⊤ D(x)=\text{embed}(c)\cdot D^{\prime}(x)^{\top} . We do not employ any normalization-based conditioning such as AdaIN Karras et al. [2019] , AdaGN Dhariwal & Nichol [2021] , Karras et al. [2022] , AdaBN Brock et al. [2018] or AdaLN Peebles & Xie [2023] for simplicity, even though they improve FID considerably.

#### Stacked MNIST.

We base this model off of the CIFAR-10 model but without class conditioning. We disable all data augmentation and shorten the burn-in phase considerably. We use a constant learning rate and did not observe any benefit of using a lower learning rate later in the training.

#### Compute resources.

We train the Stacked MNIST and CIFAR-10 models on an 8 × 8\times NVIDIA L40 node. Training took 7 hours for Stacked MNIST and 4 days for CIFAR-10. The FFHQ model was trained on an 8 × 8\times NVIDIA A6000 f0r roughly 3 weeks. The ImageNet model was trained on NVIDIA A100/H100 clusters and training took one day on 32 H100s (about 5000 H100 hours).

## Appendix E Negative Results and Future Work

Following the convention of Brock et al . Brock et al. [2018] , we report alternative design choices that did not make to our final model. Either because they failed to produce any quantitative improvement or because they would considerably complicate our minimalist design which might be better suited for future study. • We tried to apply GELU Hendrycks & Gimpel [2016] , Swish Ramachandran et al. [2017] , and SMU Biswas et al. [2021] to G G and D D and found that doing so deteriorates FID considerably. We did not try on G G xor D D . We posit two independent factors: – ConvNeXt in general does not benefit much from GELU (and possibly similar activations). Table 10 and Table 11 in Liu et al. [2022] : replacing ReLU with GELU gives little performance gain to ConvNeXt-T and virtually no performance gain to ConvNeXt-B.

– In the context of GANs, GELU and Swish have the same problem as ReLU: that they have little gradient in the negative interval. Since G is updated from the gradient of D, having these activation functions in D could sparsify the gradient of D and as a result G will not receive as much useful information from D compared to using leaky ReLU.

This does not explain the strange case of SMU Biswas et al. [2021] : SMU is a smooth approximation of leaky ReLU and does not have the sparse gradient problem. It is unclear why it also underperformed and future work awaits.

• We tried adding group normalization Wu & He [2018b] to G G and D D and it did not improve FID or training stability. We do not claim that all forms of normalizations are harmful. Our claim in principle c) only extends to normalization layers that explicitly standardizes the mean and standard deviation of the activation maps. This has been verified by prior studies Karras et al. [2020b] , Karras et al. [2023] , Sadat et al. [2024] . The harm of normalization layers extends to the adjacent field of image restoration Lim et al. [2017] , Wang et al. [2018] . To the best of our knowledge, EDM2 Karras et al. [2023] is currently the strongest diffusion UNet and it does not use normalization layers. However, it does apply normalization to the trainable weights and this improves performance considerably. We expect that applying the normalization techniques in EDM2 would improve our model’s performance.

• We tried removing the activation function after the 3 × \times 3 grouped convolution in each residual block as modern architectures Liu et al. [2022] , Yu et al. [2022] typically do not apply non-linearity after depthwise convolution. This worsened FID performance.

• We tried Pixel-Shuffle/Unshuffle Shi et al. [2016a] for changing the resolution of the activation maps, and found that without low-pass filtering, this led to high frequency artifacts similar to checkerboard artifacts even though Pixel-Shuffle does not have the uneven overlap problem that transposed convolution does. Note that bilinear resampling is equivalent to applying channel duplication/averaging with Pixel-Shuffle/Unshuffle in conjunction with a [1, 2, 1] low-pass kernel. It might be interesting in future studies to explore inplace resampling filters that apply a low-pass filtered Pixel-Shuffle/Unshuffle operation on top of a learned function that changes the number of channels.

• We tried scaling up our model size. We found that allocating more model capacity to lower resolution stages generally did not improve FID, but contributed to more rapid overfitting. Increasing model capacity at higher resolution stages always improves FID in our experiments, however scaling up higher resolution stages is very computationally expensive. Capacity distribution for each resolution stage of the model might be an important topic to explore in future studies.

• For model simplicity, we did not conduct any experiment with a transformer architecture or attention mechanism in general. We are interested to see whether adding attention blocks to a convolutional network (similar to BigGAN Brock et al. [2018] and diffusion UNet Ho et al. [2020] , Karras et al. [2022] , Karras et al. [2023] ) or using a pure transformer architecture (similar to DiT Peebles & Xie [2023] ) will result in stronger performance. Given the impressive results of EDM2 Karras et al. [2023] (which uses UNet), it seems the argument has not yet settled for generative modeling.

• We experimented with Adam β 2 = 0.999 \beta_{2}=0.999 following common practice in supervised learning and diffusion models, and found that doing so led to stability issues on our ImageNet models. We expect that introducing proper normalization to our model will resolve this problem.

• We tried mixed precision training with IEEE FP16 as this is the low precision format used in StyleGAN2-ADA Karras et al. [2020a] , StyleGAN3 Karras et al. [2021] , and EDM2 Karras et al. [2023] . This crippled the training of our model and switching to BFloat16 fixed the problem. We expect that introducing proper normalization to our model will allow us to use IEEE FP16 which offers more precision than BFloat16.

• We tried lazy regularization Karras et al. [2020b] in our early experiments where R 1 R_{1} and R 2 R_{2} were applied once every 8 minibatches. This led to slightly worse FID performance on real world datasets like FFHQ and CIFAR-10. However, it resulted in complete convergence failure on Stacked MNIST and several two dimensional toy datasets (line, circle, 25 Gaussians, etc . ), indicating potential concerns regarding the mathematical legitimacy of this trick.

## Appendix F Qualitative Results

## Appendix G Training Curves

## NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope?

Answer: [Yes]

Justification: Claim of stability is justified by Figure 2 and later experimental performance. Claim of convergence properties is justified in Appendices A,B,C. Claim of SOTA GAN is experimentally justified in Section 4 . Claims are bound to specific datasets.

Guidelines: • The answer NA means that the abstract and introduction do not include the claims made in the paper.

• The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

• The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

• It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: Please see Section 5.

Guidelines: • The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

• The authors are encouraged to create a separate "Limitations" section in their paper.

• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

• The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

• The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

• The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

• If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

• While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren’t acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: Prior knowledge of Mescheder et al . Mescheder et al. [2018] is required, but this is cited appropriately to help the reader.

Guidelines: • The answer NA means that the paper does not include theoretical results.

• All the theorems, formulas, and proofs in the paper should be numbered and cross-referenced.

• All assumptions should be clearly stated or referenced in the statement of any theorems.

• The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

• Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

• Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: Supplemental table lists all hyperparamters, and a supplemental section describes the training configurations.

Guidelines: • The answer NA means that the paper does not include experiments.