The GAN is dead; long live the GAN! A Modern GAN Baseline — part 3/7
Cleaned-text code points [31006, 46789). Bibliography omitted here; source.md includes it.
For Config E, we conduct two experiments that ablate i. i.1 (increased width with depthwise conv.) and i. i.2 (an inverted bottleneck). We add GroupedConv and reduce the bottleneck compression ratio to two given the same model size. Each bottleneck is now 1.5 × \times the width of Config A, and the FID drops to 7.51, surpassing the performance of StyleGAN2. By inverting the stem and the bottleneck dimensions to enhance the capacity of GroupedConv, our final model achieves an FID of 7.05, exceeding StyleGAN2.
### 4.2 Mode Recovery — StackedMNIST Metz et al. [2016]
We repeat the earlier experiment in 1000-mode convergence on StackedMNIST (unconditional generation), but this time with our updated architecture and with comparisons to SOTA GANs and likelihood-based methods (Tab. 5 , Fig. 6 ). One advantage brought up of likelihood-based models such as diffusion over GANs is that they achieve mode coverage Dhariwal & Nichol [2021] . We find that most GANs struggle to find all modes. But, PresGAN Dieng et al. [2019] , DDGAN Xiao et al. [2021] , and our approach are successful. Further, our method outperforms all other tested GAN models in term of KL divergence.
### 4.3 FID — FFHQ-256 Karras et al. [2019] (Optimized)
We train Config E model until convergence and with optimized hyperparameters and training schedule on FFHQ at 256 × \times 256 (unconditional generation) (Tab. 4.4 , Figs. 5 and 7 ). Please see our supplemental material for training details. Our model outperforms existing StyleGAN methods, plus four more recent diffusion-based methods. On this common dataset experimental setting, many methods (not listed here) use the bCR Zhao et al. [2021] trick—this has only been shown to improve performance on FFHQ-256 (not even at different resolutions of FFHQ) Zhao et al. [2021] , Zhang et al. [2022] . We do not use this trick.
### 4.4 FID — FFHQ-64 Karras et al. [2022]
To compare with EDM Karras et al. [2022] directly, we evaluate our model on FFHQ at 64 × \times 64 resolution. For this, we remove the two highest resolution stages of our 256 × \times 256 model, resulting in a generator that is less than half the number of parameters as EDM. Despite this, our model outperforms EDM on this dataset and needs one function evaluation only (Tab. 4.4 ).
Model NFE ↓ \downarrow FID ↓ \downarrow StyleGAN2 Karras et al. [2020b] 1 3.78 StyleGAN3-T Karras et al. [2021] 1 4.81 StyleGAN3-R Karras et al. [2021] 1 3.92 LDM Rombach et al. [2022] 200 4.98 ADM (DDIM) Dhariwal & Nichol [2021] , Lu et al. [2023] 500 8.41 ADM (DPM-Solver) Dhariwal & Nichol [2021] , Lu et al. [2023] 500 8.40 Diffusion Autoencoder Preechakul et al. [2022] , Lu et al. [2023] 500 5.81 Ours—Config E 1 2.75 With ImageNet feature leakage Kynkäänniemi et al. [2022] : PolyINR* Singh et al. [2023] 1 2.72 StyleGAN-XL* Sauer et al. [2022] 1 2.19 StyleSAN-XL* Takida et al. [2024] 1 1.68 Table 8: FFHQ-256. * denotes models that leak ImageNet features. Model NFE ↓ \downarrow FID ↓ \downarrow StyleGAN2 Karras et al. [2020b] , Lin et al. [2021] 1 3.32 MSG-GAN Karnewar & Wang [2020] , Lin et al. [2021] 1 2.7 Anycost GAN Lin et al. [2021] 1 2.52 VE Song et al. [2020] , Karras et al. [2022] 79 25.95 VP Song et al. [2020] , Karras et al. [2022] 79 3.39 EDM Karras et al. [2022] 79 2.39 Ours—Config E 1 1.95 Table 11: FFHQ-64.
### 4.5 FID — CIFAR-10 Krizhevsky et al. [2009]
We train Config E model until convergence and with optimized hyperparameters and training schedule on CIFAR-10 (conditional generation) (Tab. 12 , Fig. 9 ). Our method outperforms many other GANs by FID even though the model has relatively small capacity. For instance, StyleGAN-XL Sauer et al. [2022] has 18 M parameters in the generator and 125 M parameters in the discriminator, while our model has a 40 M parameters between the generator and discriminator combined (Fig. 4 ). Compared to diffusion models like LDM or ADM, GAN inference is significantly cheaper as it requires only one network function evaluation compared to the tens or hundreds of network function evaluations for diffusion models without distillation.
Many state-of-the-art GANs are derived from Projected GAN Sauer et al. [2021] , including StyleGAN-XL Sauer et al. [2022] and the concurrent work of StyleSAN-XL Takida et al. [2024] . These methods use a pre-trained ImageNet classifier in the discriminator. Prior work has shown that a pre-trained ImageNet discriminator can leak ImageNet features into the model Kynkäänniemi et al. [2022] , causing the model to perform better when evaluating on FID since it relies on a pre-trained ImageNet classifier for the loss. But, this does not improve results in perceptual studies Kynkäänniemi et al. [2022] . Our model produces its low FID without any ImageNet pre-training.
### 4.6 FID — ImageNet-32 Chrabaszcz et al. [2017]
We train Config E model until convergence and with optimized hyperparameters and training schedule on ImageNet-32 (conditional generation). We compare against recent GAN models and recent diffusion models in Table 4.8 . We adjust the number of parameters in the generator of our model to match StyleGAN-XL Sauer et al. [2022] ’s generator (84M parameters). Specifically, we make the model significantly wider to match. Our method achieves comparable FID despite using a 60% smaller discriminator (Tab. 4.8 ) and despite not using a pre-trained ImageNet classifier.
### 4.7 FID — ImageNet-64 Chrabaszcz et al. [2017]
We evaluate our model on ImageNet-64 to test its scalability. We stack another resolution stage on our ImageNet-32 model, resulting in a generator of 104 M parameters. This model is nearly 3 × \times smaller than diffusion-like models Dhariwal & Nichol [2021] , Karras et al. [2022] , Song et al. [2023] , Song & Dhariwal [2024] that rely on the ADM backbone, which contains about 300 M parameters. Despite the smaller model size and that our model generates samples in one step, it outperforms larger diffusion models with many NFEs on FID (Tab. 4.8 ).
### 4.8 Recall
We evaluate the recall Kynkäänniemi et al. [2019] of our model on each dataset to quantify sample diversity. In general, our model achieves a recall that is similar to or marginally worse than the diffusion model counterpart, yet superior to existing GAN models. For CIFAR-10, the recall of our model peaked at 0.57; as a point of comparison, StyleGAN-XL Sauer et al. [2022] has a worse recall of 0.47 despite its lower FID. For FFHQ, we obtain a recall of 0.53 at 64 × \times 64 and 0.49 at 256 × \times 256, whereas StyleGAN2 Karras et al. [2020b] achieved a recall of 0.43 on FFHQ-256. Our ImageNet-32 model achieved a recall of 0.63; comparable to ADM Dhariwal & Nichol [2021] . Our ImageNet-64 model achieved recall 0.59. While this is slightly worse than ≈ \approx 0.63 that many diffusion models achieve, it is better than BigGAN-deep Brock et al. [2018] which achieved a recall of 0.48.
Model NFE ↓ \downarrow FID ↓ \downarrow DDPM++ Kim et al. [2021] 1000 8.42 VDM Kingma et al. [2021] 1000 7.41 MSGAN Karnewar & Wang [2020] , Ning et al. [2023] 1 12.3 ADM Dhariwal & Nichol [2021] 1000 3.60 DDPM-IP Ning et al. [2023] 1000 2.87 Ours—Config E 1 1.27 With ImageNet feature leakage Kynkäänniemi et al. [2022] : StyleGAN-XL* Sauer et al. [2022] 1 1.10 Table 15: ImageNet-32. Model NFE ↓ \downarrow FID ↓ \downarrow BigGAN-deep Brock et al. [2018] 1 4.06 DDPM Ho et al. [2020] 250 11.0 DDIM Song et al. [2021] 50 13.7 ADM Dhariwal & Nichol [2021] § 250 2.91 EDM Karras et al. [2022] 79 2.23 CT Song et al. [2023] 2 11.1 CD Song et al. [2023] 3 4.32 iCT-deep Song & Dhariwal [2024] 2 2.77 DMD Yin et al. [2024] 1 2.62 Ours—Config E 1 2.09 With ImageNet feature leakage Kynkäänniemi et al. [2022] : StyleGAN-XL* Sauer et al. [2022] 1 1.52 Table 18: ImageNet-64. §:deterministic sampling.
## 5 Discussion and Limitations
We have shown that a simplification of GANs is possible for image generation tasks, built upon a more stable RpGAN + R 1 + R 2 +R_{1}+R_{2} objective with mathematically-demonstrated convergence properties that still provides diverse output. This stability is what lets us re-engineer a modern network architecture without the tricks of previous methods, producing the R3GAN model with competitive FID on the common datasets of Stacked-MNIST, FFHQ, CIFAR-10, and ImageNet as an empirical demonstration of the mathematical benefits.
The focus of our work is to elucidate the essential components of a minimum GAN for image generation. As such, we prioritize simplicity over functionality—we do not claim to beat the performance of every existing model on every dataset or task; merely to provide a new simple baseline that converges easily. While this makes our model a possible backbone for future GANs, it also means that it is not suitable to apply our model directly to downstream applications such as image editing or controllable generation, as our model lacks dedicated features for easy image inversion or disentangled image synthesis. For instance, we remove style injection functionality from StyleGAN even though this has a clear use. We also omitted common techniques that have been shown in previous literature to improve FID considerably. Examples include some form of adaptive normalization modulated by the latent code Dhariwal & Nichol [2021] , Karras et al. [2022] , Karras et al. [2019] , Zhang et al. [2022] , Peebles & Xie [2023] , Wang et al. [2023a] , Sahoo et al. [2023] , and using multiheaded self attention at lower resolution stages Dhariwal & Nichol [2021] , Karras et al. [2022] , Karras et al. [2023] . We aim to explore these techniques in a future study.
Further, our work is limited in its evaluation of the scalability of R3GAN models. While they show promising results on 64 × \times 64 ImageNet, we are yet to verify the scalability on higher resolution ImageNet data or large-scale text to image generation tasks Gokaslan et al. [2024] .
Finally, as a method that can improve the quality of generative models, it would be amiss not to mention that generative models—especially of people—can cause direct harm (e.g., through personalized deep fakes) and societal harm through the spread of disinformation (e.g., fake influencers).
## 6 Conclusion
This work introduced R3GAN, a new baseline GAN that features increased stability, leverages modern architectures, and does not require ad-hoc tricks that are commonplace in existing GAN models. Central to our approach is a regularized relativistic loss that provably features local convergence and that improves the stability of GAN training. This stable loss enables us to ablate various tricks that were previously necessary in GANs, and incorporate in their place modern deep architectures. The resulting streamlined baseline achieves competitive performance to SOTA models within its parameter size class. We anticipate that our backbone will help to drive future GAN research.
#### Acknowledgements.
The authors thank Xinjie Jayden Yi for contributing to the proof and Yu Cheng for helpful discussion. For compute, the authors thank Databricks Mosaic Research. Yiwen Huang was supported by a Brown University Division of Research Seed Award, and James Tompkin was supported by NSF CAREER 2144956. Volodymyr Kuleshov was supported by NSF CAREER 2145577 and NIH MIRA 1R35GM15124301.
## Appendix A Local convergence
Following Mescheder et al. [2018] , GAN training can be formulated as a dynamical system where the update operator is given by F h ( θ , ψ ) = ( θ , ψ ) + h v ( θ , ψ ) F_{h}(\theta,\psi)=(\theta,\psi)+hv(\theta,\psi) . h h is the learning rate and v v denotes the gradient vector field: v ( θ , ψ ) = ( − ∇ θ ℒ ( θ , ψ ) ∇ ψ ℒ ( θ , ψ ) ) v(\theta,\psi)=\begin{pmatrix}-\nabla_{\theta}\mathcal{L}(\theta,\psi)\\ \nabla_{\psi}\mathcal{L}(\theta,\psi)\end{pmatrix} (4) Mescheder et al. Mescheder et al. [2017] showed that local convergence near ( θ ∗ , ψ ∗ ) (\theta^{*},\psi^{*}) can be analyzed by examining the spectrum of the Jacobian J F h \textbf{J}_{F_{h}} at the equilibrium: if the Jacobian has eigenvalues with absolute value bigger than 1, then training does not converge. On the other hand, if all eigenvalues have absolute value smaller than 1, then training will converge to ( θ ∗ , ψ ∗ ) (\theta^{*},\psi^{*}) at a linear rate. If all eigenvalues have absolute value equal to 1, the convergence behavior is undetermined.
Given some calculations Mescheder et al. [2018] , we can show that the eigenvalues of the Jacobian of the update operator λ J F h \lambda_{\textbf{J}_{F_{h}}} can be determined by λ J v \lambda_{\textbf{J}_{v}} : λ J F h = 1 + h λ J v . \lambda_{\textbf{J}_{F_{h}}}=1+h\lambda_{\textbf{J}_{v}}\ . (5) That is, given small enough h h Mescheder et al. [2018] , the training dynamics can instead be examined using λ J v \lambda_{\textbf{J}_{v}} , i.e . , the eigenvalues of the Jacobian of the gradient vector field. If all λ J v \lambda_{\textbf{J}_{v}} have a negative real part, the training will locally converge to ( θ ∗ , ψ ∗ ) (\theta^{*},\psi^{*}) at a linear rate. On the other hand, if some λ J v \lambda_{\textbf{J}_{v}} have a positive real part, the training is not convergent. If all λ J v \lambda_{\textbf{J}_{v}} have a zero real part, the convergence behavior is inconclusive.
## Appendix B DiracRpGAN: A demonstration of non-convergence
#### Summary.
To obtain DiracRpGAN, we apply Eq. 2 to the DiracGAN Mescheder et al. [2018] problem setting. After simplification, DiracRpGAN and DiracGAN are different only by a constant. They have the same gradient vector field, therefore all proofs are identical to Mescheder et al . Mescheder et al. [2018] .
#### Definition B.1.
The DiracRpGAN consists of a (univariate) generator distribution p θ = δ θ p_{\theta}=\delta_{\theta} and a linear discriminator D ψ ( x ) = ψ ⋅ x D_{\psi}(x)=\psi\cdot x . The true data distribution p 𝒟 p_{\mathcal{D}} is given by a Dirac distribution concentrated at 0.
In this setup, the RpGAN training objective is given by: ℒ ( θ , ψ ) = f ( ψ θ ) . \mathcal{L}(\theta,\psi)=f(\psi\theta)\ . (6) We can now show analytically that DiracRpGAN does not converge without regularzation.
#### Lemma B.2.
The unique equilibrium point of the training objective in Eq. 6 is given by θ = ψ = 0 \theta=\psi=0 . Moreover, the Jacobian of the gradient vector field at the equilibrium point has the two eigenvalues ± f ′ ( 0 ) i \pm f^{\prime}(0)i which are both on the imaginary axis.
The gradient vector field v v of Eq. 6 is given by: v ( θ , ψ ) = ( − ∇ θ ℒ ( θ , ψ ) ∇ ψ ℒ ( θ , ψ ) ) = ( − ψ f ′ ( ψ θ ) θ f ′ ( ψ θ ) ) v(\theta,\psi)=\begin{pmatrix}-\nabla_{\theta}\mathcal{L}(\theta,\psi)\\ \nabla_{\psi}\mathcal{L}(\theta,\psi)\end{pmatrix}=\begin{pmatrix}-\psi f^{\prime}(\psi\theta)\\ \theta f^{\prime}(\psi\theta)\end{pmatrix} (7) and the Jacobian of v v : J v = ( − ψ 2 f ′′ ( ψ θ ) − f ′ ( ψ θ ) − ψ θ f ′′ ( ψ θ ) f ′ ( ψ θ ) + ψ θ f ′′ ( ψ θ ) θ 2 f ′′ ( ψ θ ) ) \textbf{J}_{v}=\begin{pmatrix}-\psi^{2}f^{\prime\prime}(\psi\theta)&-f^{\prime}(\psi\theta)-\psi\theta f^{\prime\prime}(\psi\theta)\\ f^{\prime}(\psi\theta)+\psi\theta f^{\prime\prime}(\psi\theta)&\theta^{2}f^{\prime\prime}(\psi\theta)\end{pmatrix} (8) Evaluating J v \textbf{J}_{v} at the equilibrium point θ = ψ = 0 \theta=\psi=0 gives us: J v | ( 0 , 0 ) = ( 0 − f ′ ( 0 ) f ′ ( 0 ) 0 ) \textbf{J}_{v}\biggr\rvert_{(0,0)}=\begin{pmatrix}0&-f^{\prime}(0)\\ f^{\prime}(0)&0\end{pmatrix} (9) Therefore, the eigenvalues of J v \textbf{J}_{v} are λ 1 / 2 = ± f ′ ( 0 ) i \lambda_{1/2}=\pm f^{\prime}(0)i , both of which have a real part of 0. Thus, the convergence of DiracRpGAN is inconclusive and further analysis is required.
#### Lemma B.3.