The GAN is dead; long live the GAN! A Modern GAN Baseline — part 2/7

Cleaned-text code points [15369, 31006). Bibliography omitted here; source.md includes it.

Figure 2: Generator G G loss for different objectives over training. Regardless of which objective is used, training diverges with only R 1 R_{1} and succeeded with both R 1 R_{1} and R 2 R_{2} . Convergence failure with only R 1 R_{1} was noted by Lee et al. Lee et al. [2021] . Loss # modes ↑ \uparrow D KL D_{\text{KL}} ↓ \downarrow RpGAN + R 1 + R 2 +R_{1}+R_{2} 𝟏𝟎𝟎𝟎 \mathbf{1000} 0.0781 \mathbf{0.0781} GAN + R 1 + R 2 +R_{1}+R_{2} 693 693 0.9270 0.9270 RpGAN + R 1 +R_{1} Fail Fail GAN + R 1 +R_{1} Fail Fail Table 3: StackedMNIST Lin et al. [2018] result for each loss function. The maximum possible mode coverage is 1000. “Fail” indicates that training diverged early on.

A conventional GAN loss with R 1 R_{1} , as used by Mescheder et al. Mescheder et al. [2018] and the StyleGAN series Karras et al. [2019] , Karras et al. [2020b] , Karras et al. [2021] , diverges quickly (Fig. 2 ). Next, while theoretically sufficient for local convergence, RpGAN with only R 1 R_{1} regularization is also unstable and diverges quickly 2 2 2 Varying γ \gamma from 0.1 to 100 does not stabilize training. . In each case, the gradient of D D on fake samples explodes when training diverges. With both R 1 R_{1} and R 2 R_{2} , training becomes stable for both the classic GAN and RpGAN. Now stable, we can see that the classic GAN suffers from mode dropping, whereas RpGAN achieves full mode coverage (Tab. 2 ) and reduces D KL D_{\text{KL}} from 0.9270 to 0.0781. As a point of contrast, StyleGAN Karras et al. [2019] , Karras et al. [2020b] , Karras et al. [2020a] , Karras et al. [2021] uses the minibatch standard deviation trick to reduce mode dropping, improving mode coverage from 857 to 881 on StackedMNIST 3 3 3 These numbers are from Karras et al . Karras et al. [2017] , Table 4. ”857” corresponds to a low-capacity version of a progressive GAN and ”881” adds the minibatch standard deviation trick. Further comparisons via loss curves are difficult since progressive GAN is a substantially different model than the small ResNet we use for this experiment. and with barely any improvement on D KL D_{\text{KL}} Karras et al. [2017] .

R 1 R_{1} alone is not sufficient for globally-convergent training. While a theoretical analysis of this is difficult, our small demonstration still provides insights into the assumptions of our convergence proof. In particular, the assumption that ( θ , ψ ) (\theta,\psi) are sufficiently close to ( θ ∗ , ψ ∗ ) (\theta^{*},\psi^{*}) is highly unlikely early in training. In this scenario, if D D is sufficiently powerful, regularizing D D solely on real data is not likely to have much effect on D D ’s behavior on fake data and so training can fail due to an ill-behaved D D on fake data. This observation has been made by previous studies Thanh-Tung et al. [2019] , Tao & Wang [2020] specifically for empirical GAN training, that regularizing an empirical discriminator with only R 1 R_{1} leads to gradient explosion on fake data due to the memorization of real samples.

Thus, the practical solution is to regularize D D on both real and fake data. The benefit of doing so can be viewed from the insight of Roth et al . Roth et al. [2017] : that applying R 1 R_{1} and R 2 R_{2} in conjunction smooths both p 𝒟 p_{\mathcal{D}} and p θ p_{\theta} which makes learning easier than only smoothing p 𝒟 p_{\mathcal{D}} . We also find empirically that with both R 1 R_{1} and R 2 R_{2} in place, D D tends to satisfy 𝔼 x ∼ p 𝒟 ​ [ ‖ ∇ x D ‖ 2 ] ≈ 𝔼 x ∼ p θ ​ [ ‖ ∇ x D ‖ 2 ] \mathbb{E}_{x\sim p_{\mathcal{D}}}\left[\left\|\nabla_{x}D\right\|^{2}\right]\approx\mathbb{E}_{x\sim p_{\theta}}\left[\left\|\nabla_{x}D\right\|^{2}\right] even early in the training. Jolicoeur-Martineau et al . Jolicoeur-Martineau & Mitliagkas [2019] show that in this case D D becomes a maximum margin classifier—but if only one regularization term is applied, this does not hold. Additionally, having roughly the same gradient norm on real and fake data potentially reduces discriminator overfitting, as Fang et al . Fang et al. [2022] observe that the gradient norm on real and fake data diverges when D D starts to overfit.

## 3 A Roadmap to a New Baseline — R3GAN

The well-behaved RpGAN + R 1 R_{1} + R 2 R_{2} loss alleviates GAN optimization problems, and lets us proceed to build a minimalist baseline—R3GAN—with recent network backbone advances in mind Liu et al. [2022] , Yu et al. [2022] . Rather than simply state the new approach, we will draw out a roadmap from the StyleGAN2 baseline Karras et al. [2020a] . This model (Config A; identical to Karras et al. [2020a] ) consists of a VGG-like Simonyan & Zisserman [2014] backbone for G G , a ResNet D D , a few techniques that facilitate style-based generation, and many tricks that serve as patches to the weak backbone. Then, we remove all non-essential features of StyleGAN2 (Config B), apply our loss function (Config C), and gradually modernize the network backbone (Config D-E).

We evaluate each configuration on FFHQ 256 × 256 256\times 256 Karras et al. [2019] . Network capacity is kept roughly the same for all configurations—both G G and D D have about 25 M trainable parameters. Each configuration is trained until D D sees 5 M real images. We inherit training hyperparameters ( e.g . , optimizer settings, batch size, EMA decay length) from Config A unless otherwise specified. We tune the training hyperparameters for our final model and show the converged result in Sec. 4 .

#### Minimum baseline (Config B).

We strip away all StyleGAN2 features, retaining only the raw network backbone and basic image generation capability. The features fall into three categories: • Style-based generation: mapping network Karras et al. [2019] , style injection Karras et al. [2019] , weight modulation/demodulation Karras et al. [2020b] , noise injection Karras et al. [2019] .

• Image manipulation enhancements: mixing regularization Karras et al. [2019] , path length regularization Karras et al. [2020b] .

• Tricks: z z normalization Karras et al. [2017] , minibatch stddev Karras et al. [2017] , equalized learning rate Karras et al. [2017] , lazy regularization Karras et al. [2020b] .

Following Sauer et al. [2022] , Sauer et al. [2023] , we reduce the dimension of z z to 64. The absence of equalized learning rate necessitates a lower learning rate, reduced from 2.5 × \times 10 -3 to 5 × \times 10 -5 . Despite a higher FID of 12.46 than Config-A, this simplified baseline produces reasonable sample quality and stable training. We compare this with DCGAN Radford et al. [2015] , an early attempt at image generation. Key differences include: a) Convergent training objective with R 1 R_{1} regularization.

b) Smaller learning rate, avoiding momentum optimizer (Adam β 1 = 0 \beta_{1}=0 ).

c) No normalization layer in G G or D D .

d) Proper resampling via bilinear interpolation instead of strided (transposed) convolution.

e) Leaky ReLU in both G G and D D , no tanh in the output layer of G G .

f) 4 × \times 4 constant input for G G , output skips for G G , ResNet D D .

Experimental findings from StyleGAN. Violating a) , b) , or c) often leads to training failures. Gidel et al . Gidel et al. [2019] show that negative momentum can improve GAN training dynamics. Since optimal negative momentum is another challenging hyperparameter, we do not use any momentum to avoid worsening GAN training dynamics. Studies suggest normalization layers harm generative models Karras et al. [2020b] , Karras et al. [2023] . Batch normalization Ioffe & Szegedy [2015] often cripples training due to dependencies across multiple samples, and is incompatible with R 1 R_{1} , R 2 R_{2} , or RpGAN that assume independent handling of each sample. Weaker data-independent normalizations Karras et al. [2020b] , Karras et al. [2023] might help; we leave this for future work. Early GANs may succeed despite violating a) and c) , possibly constituting a full-rank solution Mescheder et al. [2018] to Eq. 1 .

Violations of d) or e) do not significantly impair training stability but negatively affect sample quality. Improper transposed convolution can cause checkerboard artifacts, unresolved even with subpixel convolution Shi et al. [2016b] or carefully tuned transposed convolution unless a low-pass filter is applied. Interpolation methods avoid this issue, varying from nearest neighbor Karras et al. [2017] to Kaiser filters Karras et al. [2021] . We use bilinear interpolation for simplicity. For activation functions, smooth approximations of (leaky) ReLU, such as Swish Ramachandran et al. [2017] , GELU Hendrycks & Gimpel [2016] , and SMU Biswas et al. [2021] , worsen FID. PReLU He et al. [2015] marginally improves FID but increases VRAM usage, so we use leaky ReLU.

All subsequent configurations adhere to a) through e) . Violation of f) is acceptable as it pertains to the network backbone of StyleGAN2 Karras et al. [2020b] , modernized in Config D and E.

#### Well-behaved loss function (Config C).

We use the loss function proposed in Section 2 and this reduces FID to 11.65. We hypothesize that the network backbone in Config B is the limiting factor.

(a) Overall view (b) StyleGAN2 architecture blocks Karras et al. [2020b] (Config A) (c) Ours (Config E)

#### General network modernization (Config D).

First, we apply the 1-3-1 bottleneck ResNet architecture He et al. [2016a] , He et al. [2016b] to both G G and D D . This is the direct ancestor of all modern vision backbones Liu et al. [2022] , Yu et al. [2022] . We also incorporate principles discovered in Config B and various modernization efforts from ConvNeXt Liu et al. [2022] . We categorize the roadmap of ConvNeXt as follows:

i. Consistently beneficial: i.1) increased width with depthwise convolution i.2) inverted bottleneck i.3) fewer activation functions i.4) separate resampling layers.

ii. Negligible performance gain: ii.1) large kernel depthwise conv. with fewer channels ii.2) swap ReLU with GELU ii.3) fewer normalization layers ii.4) swap batch norm. with layer norm.

iii. Irrelevant to our setting: iii.1) improved training recipe iii.2) stage ratio iii.3) ‘patchify’ stem.

We aim to apply i) to our model, specifically i. i.3 and i. i.4 for the classic ResNet, while reserving i. i.1 and i. i.2 for Config E. Many aspects of ii) were introduced merely to mimic vision transformers Liu et al. [2021] , Dosovitskiy et al. [2020] without yielding significant improvements Liu et al. [2022] . ii. ii.3 and ii. ii.4 are inapplicable due to our avoidance of normalization layers following principle c) . ii. ii.2 contradicts our finding that GELU deteriorates GAN performance, thus we use leaky ReLU per principle e) . Liu et al . emphasize large conv. kernels (ii. ii.1 ) Liu et al. [2022] , but this results in slightly worse performance compared to wider 3 × \times 3 conv. layers, so we do not adopt this ConvNeXt design choice.

#### Neural network architecture details.

Given i. i.3 , i. i.4 , and principles c) , d) , and e) , we can replace the StyleGAN2 backbone with a modernized ResNet. We use a fully symmetric design for G G and D D with 25 M parameters each, comparable to Config-A. The architecture is minimalist: each resolution stage has one transition layer and two residual blocks. The transition layer consists of bilinear resampling and an optional 1 × \times 1 conv. for changing spatial size and feature map channels. The residual block includes five operations: Conv1 × \times 1 → \rightarrow Leaky ReLU → \rightarrow Conv3 × \times 3 → \rightarrow Leaky ReLU → \rightarrow Conv1 × \times 1, with the final Conv1 × \times 1 having no bias term. For the 4 × \times 4 resolution stage, the transition layer is replaced by a basis layer for G G and a classifier head for D D . The basis layer, similar to StyleGAN Karras et al. [2019] , Karras et al. [2020b] , uses 4 × \times 4 learnable feature maps modulated by z z via a linear layer. The classifier head uses a global 4 × \times 4 depthwise conv. to remove spatial extent, followed by a linear layer to produce D D ’s output. We maintain the width ratio for each resolution stage as in Config A, making the stem width 3 × \times as wide due to the efficient 1 × \times 1 conv. The 3 × \times 3 conv. in the residual block has a compression ratio of 4, following He et al. [2016a] , He et al. [2016b] , making the bottleneck width 0.75 × \times as wide as Config A.

To avoid variance explosion due to the lack of normalization, we employ fix-up initialization Zhang et al. [2019] : We zero-initialize the last convolutional layer in each residual block and scale down the initialization of the other two convolutional layers in the block by L − 0.25 L^{-0.25} , where L L is the number of residual blocks. We avoid other fix-up tricks, such as excessive bias terms and a learnable multiplier.

#### Bottleneck modernization (Config E).

Now that we have settled on the overall architecture, we investigate how the residual block can be modernized, specifically i. i.1 ) and i. i.2 ). First, we explore i. i.1 and replace the 3 × \times 3 convolution in the residual block with a grouped convolution. We set the group size to 16 rather than 1 ( i.e . depthwise convolution as in ConvNeXt) as depthwise convolution is highly inefficient on GPUs and is not much faster than using a larger group size. With grouped convolution, we can reduce the bottleneck compression ratio to two given the same model size. This increases the width of the bottleneck to 1.5 × \times as wide as Config A. Finally, we notice that the compute cost of grouped convolution is negligible compared to 1 × \times 1 convolution, and so we seek to enhance the capacity of grouped convolution. We apply i. i.2 ), which inverts the bottleneck width and the stem width, and which doubles the width of grouped convolutions without any increase in model size. Figure 3 depicts our final design, which reflects modern CNN architectures.

## 4 Experiments Details

### 4.1 Roadmap Insights on FFHQ-256 Karras et al. [2019]

As per Table 4 , Config A (vanilla StyleGAN2) achieves an FID of 7.52 using the official implementation on FFHQ-256. Config B with all tricks removed achieves an FID of 12.46—performance drops as expected. Config C, with a well-behaved loss, achieves an FID of 11.65. But, now training is sufficiently stable to improve the architecture.

Config D, which improves G G and D D based on the classic ResNet and ConvNeXt findings, achieves an FID of 9.95. The output skips of the StyleGAN2 generator are no longer useful given our new architecture; including them produces a worse FID of 10.17. Karras et al . find that the benefit of output skips is mostly related to gradient magnitude dynamics Karras et al. [2021] , and this has been addressed by our ResNet architecture. For StyleGAN2, Karras et al . conclude that a ResNet architecture is harmful to G G Karras et al. [2020b] , but this is not true in our case as their ResNet implementation is considerably different from ours: 1) Karras et al . use one 3-3 residual block for each resolution stage, while we have a separate transition layer and two 1-3-1 residual blocks; 2) i.3) and i.4) are violated as they do not have a linear residual block Sandler et al. [2018] and the transition layer is placed on the skip branch of the residual block rather than the stem; 3) the essential principle of ResNet He et al. [2016a] —identity mapping He et al. [2016b] —is violated as Karras et al . divide the output of the residual block by 2 \sqrt{2} to avoid variance explosion due to the absence of a proper initialization scheme.