The GAN is dead; long live the GAN! A Modern GAN Baseline — part 1/7

Cleaned-text code points [0, 15369). Bibliography omitted here; source.md includes it.

# The GAN is dead; long live the GAN! A Modern Baseline GAN

###### Abstract

There is a widely-spread claim that GANs are difficult to train, and GAN architectures in the literature are littered with empirical tricks. We provide evidence against this claim and build a modern GAN baseline in a more principled manner. First, we derive a well-behaved regularized relativistic GAN loss that addresses issues of mode dropping and non-convergence that were previously tackled via a bag of ad-hoc tricks. We analyze our loss mathematically and prove that it admits local convergence guarantees, unlike most existing relativistic losses. Second, this loss allows us to discard all ad-hoc tricks and replace outdated backbones used in common GANs with modern architectures. Using StyleGAN2 as an example, we present a roadmap of simplification and modernization that results in a new minimalist baseline—R3GAN (“Re-GAN”). Despite being simple, our approach surpasses StyleGAN2 on FFHQ, ImageNet, CIFAR, and Stacked MNIST datasets, and compares favorably against state-of-the-art GANs and diffusion models. Code: https://www.github.com/brownvc/R3GAN

## 1 Introduction

Generative adversarial networks (GANs) let us generate high-quality images in a single forward pass. However, the original objective in Goodfellow et al . Goodfellow et al. [2020] , is notoriously difficult to optimize due to its minimax nature. This leads to a fear that training might diverge at any point due to instability, and a fear that generated images might lose diversity through mode collapse. While there has been progress in GAN objectives Gulrajani et al. [2017] , Jolicoeur-Martineau [2018] , Sun et al. [2020] , Mescheder et al. [2018] , Roth et al. [2017] , practically, the effects of brittle losses are still regularly felt. This notoriety has had a lasting negative impact on GAN research.

A complementary issue—partly motivated by this instability—is that existing popular GAN backbones like StyleGAN Karras et al. [2019] , Karras et al. [2020b] , Karras et al. [2020a] , Karras et al. [2021] use many poorly-understood empirical tricks with little theory. For instance, StyleGAN uses a gradient penalized non-saturating loss Mescheder et al. [2018] to increase stability (affecting sample diversity), but then employs a minibatch standard deviation trick Karras et al. [2017] to increase sample diversity. Without tricks, the StyleGAN backbone still resembles DCGAN Radford et al. [2015] from 2015, yet it is still the common backbone of SOTA GANs such as GigaGAN Kang et al. [2023b] and StyleGAN-T Sauer et al. [2023] . Advances in GANs have been conservative compared to other generative models such as diffusion models Ho et al. [2020] , Song et al. [2020] , Karras et al. [2022] , Karras et al. [2023] , where modern computer vision techniques such as multi-headed self attention Vaswani et al. [2017] and backbones such as preactivated ResNet He et al. [2016b] , U-Net Ronneberger et al. [2015] and vision transformers (ViTs) Dosovitskiy et al. [2020] are the norm. Given outdated backbones, it is not surprising that there is a widely-spread belief that GANs do not scale in terms of quantitative metrics like Frechet Inception Distance Heusel et al. [2017] .

We reconsider this situation: we show that by combining progress in objectives into a regularized training loss, GANs gain improved training stability, which allows us to upgrade GANs with modern backbones. First, we propose a novel objective that augments the relativistic pairing GAN loss (RpGAN; Jolicoeur-Martineau [2018] ) with zero-centered gradient penalties Mescheder et al. [2018] , Roth et al. [2017] , improving stability Gulrajani et al. [2017] , Roth et al. [2017] , Mescheder et al. [2018] . We show mathematically that gradient-penalized RpGAN enjoys the same guarantee of local convergence as regularized classic GANs, and that removing our regularization scheme induces non-convergence.

Once we have a well-behaved loss, none of the GAN tricks are necessary Karras et al. [2017] , Karras et al. [2020b] , and we are free to engineer a modern SOTA backbone architecture. We strip StyleGAN of all its features, identify those that are essential, then borrow new architecture designs from modern ConvNets and transformers Liu et al. [2022] , Yu et al. [2022] . Briefly, we find that proper ResNet design He et al. [2016b] , Sandler et al. [2018] , initialization Zhang et al. [2019] , and resampling Karras et al. [2019] , Karras et al. [2020b] , Karras et al. [2021] , Zhang [2019] are important, along with grouped convolution Xie et al. [2017] , Chollet [2017] and no normalization Karras et al. [2020b] , Karras et al. [2023] , Gulrajani et al. [2017] , Wang et al. [2018] , Brock et al. [2021] . This leads to a design that is simpler than StyleGAN and improves FID performance for the same network capacity (2.75 vs. 3.78 on FFHQ-256).

In summary, our work first argues mathematically that GANs need not be tricky to train via an improved regularized loss. Then, it empirically develops a simple GAN baseline that, without any tricks, compares favorably by FID to StyleGAN Karras et al. [2019] , Karras et al. [2020b] , Karras et al. [2021] , other SOTA GANs Brock et al. [2018] , Lee et al. [2021] , Xiao et al. [2021] , and diffusion models Ho et al. [2020] , Song et al. [2020] , Vahdat et al. [2021] across FFHQ, ImageNet, CIFAR, and Stacked MNIST datasets.

## 2 Serving Two Masters: Stability and Diversity with RpGAN + R 1 + R 2 +R_{1}+R_{2}

In defining a GAN objective, we tackle two challenges: stability and diversity. Some previous work deals with stability Karras et al. [2019] , Karras et al. [2020b] , Karras et al. [2021] and other previous work deals with mode collapse Jolicoeur-Martineau [2018] . To make progress in both, we combine a stable method with a simple regularizer that is grounded by theory.

### 2.1 Traditional GAN

A traditional GAN Goodfellow et al. [2020] , Nowozin et al. [2016] is formulated as a minimax game between a discriminator (or critic) D ψ D_{\psi} and a generator G θ G_{\theta} . Given real data x ∼ p 𝒟 x\sim p_{\mathcal{D}} and fake data x ∼ p θ x\sim p_{\theta} produced by G θ G_{\theta} , the most general form of a GAN is given by: ℒ ⁡ ( θ , ψ ) = 𝔼 z ∼ p z ​ [ f ⁡ ( D ψ ​ ( G θ ​ ( z ) ) ) ] + 𝔼 x ∼ p 𝒟 ​ [ f ⁡ ( − D ψ ​ ( x ) ) ] \displaystyle\begin{split}\mathcal{L}(\theta,\psi)=\mathbb{E}_{z\sim p_{z}}\left[f\left(D_{\psi}(G_{\theta}(z))\right)\right]+\mathbb{E}_{x\sim p_{\mathcal{D}}}\left[f\left(-D_{\psi}(x)\right)\right]\end{split} (1) where G G tries to minimize ℒ \mathcal{L} while D D tries to maximize it. The choice of f f is flexible Mao et al. [2017] , Lim & Ye [2017] . In particular, f ⁡ ( t ) = − log ⁡ ( 1 + e − t ) f(t)=-\log(1+e^{-t}) recovers the classic GAN by Goodfellow et al . Goodfellow et al. [2020] . For the rest of this work, this will be our choice of f f Nowozin et al. [2016] .

It has been shown that Equation 1 has convex properties when p θ p_{\theta} can be optimized directly Goodfellow et al. [2020] , Sun et al. [2020] . However, in practical implementations, the empirical GAN loss typically shifts fake samples beyond the decision boundary set by D D , as opposed to directly updating the density function p θ p_{\theta} . This deviation leads to a significantly more challenging problem, characterized by susceptibility to two prevalent failure scenarios: mode collapse/dropping 1 1 1 While mode collapse and mode dropping are technically distinct issues, they are used interchangeably in this context to describe the common problem where supp ⁡ ( p θ ) \supp(p_{\theta}) does not comprehensively cover supp ⁡ ( p 𝒟 ) \supp(p_{\mathcal{D}}) . Mode collapse refers to the generator producing a limited diversity of samples (i.e., one image for the entire distribution), whereas mode dropping involves the generator failing to represent certain modes of the data distribution (ignoring entire subsets of the training distribution). and non-convergence.

### 2.2 Relativistic f f -GAN

We employ a slightly different minimax game named relativistic pairing GAN (RpGAN) by Jolicoeur-Martineau et al . Jolicoeur-Martineau [2018] to address mode dropping. The general RpGAN is defined as: ℒ ⁡ ( θ , ψ ) = 𝔼 z ∼ p z x ∼ p 𝒟 ​ [ f ⁡ ( D ψ ​ ( G θ ​ ( z ) ) − D ψ ​ ( x ) ) ] \mathcal{L}(\theta,\psi)=\mathbb{E}_{\begin{subarray}{c}z\sim p_{z}\\ x\sim p_{\mathcal{D}}\end{subarray}}\left[f\left(D_{\psi}(G_{\theta}(z))-D_{\psi}(x)\right)\right] (2) Although Eq. 2 differs only slightly from Eq. 1 , evaluating this critic difference has a fundamental impact on the landscape of ℒ \mathcal{L} . Since Eq. 1 merely requires D D to separate real and fake data, in the scenario where all real and fake data can be separated by a single decision boundary, the empirical GAN loss encourages G G to simply move all fake samples barely past this single boundary—this degenerate solution is what we observe as mode collapse/dropping. Sun et al . Sun et al. [2020] characterize such degenerate solutions as bad local minima in the landscape of ℒ \mathcal{L} , and show that Eq. 1 has exponentially many bad local minima. The culprit is the existence of a single decision boundary that naturally arises when real and fake data are considered in isolation. RpGAN introduces a simple solution by coupling real and fake data, i.e . a fake sample is critiqued by its realness relative to a real sample, which effectively maintains a decision boundary in the neighborhood of each real sample and hence forbids mode dropping. Sun et al . Sun et al. [2020] show that the landscape of Eq. 2 contains no local minima that correspond to mode dropping solutions, and that every basin is a global minimum.

### 2.3 Training Dynamics of RpGAN

Although the RpGAN landscape result Sun et al. [2020] allows us to address mode dropping, the training dynamics of RpGAN have yet to be studied. The ultimate goal of Eq. 2 is to find the equilibrium ( θ ∗ , ψ ∗ ) (\theta^{*},\psi^{*}) such that p θ ∗ = p 𝒟 p_{\theta^{*}}=p_{\mathcal{D}} and D ψ ∗ D_{\psi^{*}} is constant everywhere on p 𝒟 p_{\mathcal{D}} . Sun et al . Sun et al. [2020] show that θ ∗ \theta^{*} is globally reachable along a non-increasing trajectory in the landscape of Eq. 2 under reasonable assumptions. However, the existence of such a trajectory does not necessarily mean that gradient descent will find it. Jolicoeur-Martineau et al . show empirically that unregularized RpGAN does not perform well Jolicoeur-Martineau [2018] .

Proposition I. (Informal) Unregularized RpGAN does not always converge using gradient descent.

We confirm this proposition with a proof in Appendix B. We show analytically that RpGAN does not converge for certain types of p 𝒟 p_{\mathcal{D}} , such as ones that approach a delta distribution. Thus, further regularization is necessary to fill in the missing piece of a well-behaved loss.

#### Zero-centered gradient penalties.

To tackle RpGAN non-convergence, we explore gradient penalties as the solution since it is proven that zero-centered gradient penalties (0-GP) facilitate convergent training for classic GANs Mescheder et al. [2018] . The two most commonly-used 0-GPs are R 1 R_{1} and R 2 R_{2} : R 1 ​ ( ψ ) \displaystyle R_{1}(\psi) = γ 2 ​ 𝔼 x ∼ p 𝒟 ​ [ ‖ ∇ x D ψ ‖ 2 ] \displaystyle=\frac{\gamma}{2}\mathbb{E}_{x\sim p_{\mathcal{D}}}\left[\left\|\nabla_{x}D_{\psi}\right\|^{2}\right] (3) R 2 ​ ( θ , ψ ) \displaystyle R_{2}(\theta,\psi) = γ 2 ​ 𝔼 x ∼ p θ ​ [ ‖ ∇ x D ψ ‖ 2 ] \displaystyle=\frac{\gamma}{2}\mathbb{E}_{x\sim p_{\theta}}\hskip 1.70709pt\left[\left\|\nabla_{x}D_{\psi}\right\|^{2}\right] R 1 R_{1} penalizes the gradient norm of D D on real data, and R 2 R_{2} penalizes the gradient norm of D D on fake data. Analysis on the training dynamics of GANs has thus far focused on local convergence Nagarajan & Kolter [2017] , Mescheder et al. [2017] , Mescheder et al. [2018] , i.e . , whether the training at least converges when ( θ , ψ ) (\theta,\psi) are in a neighborhood of ( θ ∗ , ψ ∗ ) (\theta^{*},\psi^{*}) . In such a scenario, the convergence behavior can be analyzed Nagarajan & Kolter [2017] , Mescheder et al. [2017] , Mescheder et al. [2018] by examining the spectrum of the Jacobian of the gradient vector field ( − ∇ θ ℒ , ∇ ψ ℒ ) \left(-\nabla_{\theta}\mathcal{L},\nabla_{\psi}\mathcal{L}\right) at ( θ ∗ , ψ ∗ ) (\theta^{*},\psi^{*}) . The key insight here is that when G G already produces the true distribution, we want ∇ x D = 0 \nabla_{x}D=0 , so that G G is not pushed away from its optimal state, and thus the training does not oscillate. R 1 R_{1} and R 2 R_{2} impose such a constraint when p θ = p 𝒟 p_{\theta}=p_{\mathcal{D}} . This also explains why earlier attempts at gradient penalties, such as the one-centered gradient penalty (1-GP) in WGAN-GP Gulrajani et al. [2017] , fail to achieve convergent training Mescheder et al. [2018] as they still encourage D D to have a non-zero slope when G G has reached optimality.

Since the same insight also applies to RpGAN, we extend our previous analysis and show that:

Proposition II. (Informal) RpGAN with R 1 R_{1} or R 2 R_{2} regularization is locally convergent subject to similar assumptions as in Mescheder et al . Mescheder et al. [2018] .

In Appendix C, our proof similarly analyzes the eigenvalues of the Jacobian of the regularized RpGAN gradient vector field at ( θ ∗ , ψ ∗ ) (\theta^{*},\psi^{*}) . We show that all eigenvalues have a negative real part; thus, regularized RpGAN is convergent in a neighborhood of ( θ ∗ , ψ ∗ ) (\theta^{*},\psi^{*}) for small enough learning rates Mescheder et al. [2018] .

#### Discussion.

Another line of work Roth et al. [2017] links R 1 R_{1} and R 2 R_{2} to instance noise Sønderby et al. [2016] as its analytical approximation. Roth et al. Roth et al. [2017] showed that for the classic GAN Goodfellow et al. [2020] by Goodfellow et al . , R 1 R_{1} approximates convolving p 𝒟 p_{\mathcal{D}} with the density function of 𝒩 ⁡ ( 0 , γ ​ I ) \mathcal{N}(0,\gamma I) , up to additional weighting and a Laplacian error term. R 2 R_{2} likewise approximates convolving p θ p_{\theta} with 𝒩 ⁡ ( 0 , γ ​ I ) \mathcal{N}(0,\gamma I) up to similar error terms. The Laplacian error terms from R 1 R_{1} , R 2 R_{2} cancel when D ψ D_{\psi} approaches D ψ ∗ D_{\psi^{*}} . We do not extend Roth et al . ’s proof Roth et al. [2017] to RpGAN; however, this approach might provide complimentary insights to our work, which follows the strategy of Mescheder et al . Mescheder et al. [2018] .

### 2.4 A Practical Demonstration

We experiment with how well-behaved our loss is on StackedMNIST Lin et al. [2018] which consists of 1000 uniformly-distributed modes. The network is a small ResNet He et al. [2016b] for G G and D D without any normalization layers Ioffe & Szegedy [2015] , Wu & He [2018a] , Ba et al. [2016] , Ulyanov et al. [2016] . Through the use of a pretrained MNIST classifier, we can explicitly measure how many modes of p 𝒟 p_{\mathcal{D}} are recovered by p θ p_{\theta} . Furthermore, we can estimate the reverse KL divergence between the fake and real samples D KL ( p θ ∥ p 𝒟 ) D_{\text{KL}}\left(p_{\theta}\parallel p_{\mathcal{D}}\right) via the KL divergence between the categorical distribution of p θ p_{\theta} and the true uniform distribution.