The GAN is dead; long live the GAN! A Modern GAN Baseline — part 4/7
Cleaned-text code points [46789, 62595). Bibliography omitted here; source.md includes it.
The integral curves of the gradient vector field v ( θ , ψ ) v(\theta,\psi) do not converge to the equilibrium point. More specifically, every integral curve ( θ ( t ) , ψ ( t ) ) (\theta(t),\psi(t)) of the gradient vector field v ( θ , ψ ) v(\theta,\psi) satisfies θ ( t ) 2 + ψ ( t ) 2 = c o n s t \theta(t)^{2}+\psi(t)^{2}=const for all t ∈ [ 0 , ∞ ) t\in[0,\infty) .
Let R ( θ , ψ ) = 1 2 ( θ 2 + ψ 2 ) R(\theta,\psi)=\frac{1}{2}(\theta^{2}+\psi^{2}) , then: d d t R ( θ ( t ) , ψ ( t ) ) \displaystyle\frac{\mathrm{d}}{\mathrm{d}t}R(\theta(t),\psi(t)) = − θ ( t ) ψ ( t ) f ′ ( θ ( t ) ψ ( t ) ) + ψ ( t ) θ ( t ) f ′ ( θ ( t ) ψ ( t ) ) \displaystyle=-\theta(t)\psi(t)f^{\prime}(\theta(t)\psi(t))+\psi(t)\theta(t)f^{\prime}(\theta(t)\psi(t)) = 0 . \displaystyle=0\ . (10) We see that the distance between ( θ , ψ ) (\theta,\psi) and the equilibrium point ( 0 , 0 ) (0,0) stays constant. Therefore, training runs in circles and never converges.
Next, we investigate the convergence behavior of DiracRpGAN with regularization. For DiracRpGAN, both R 1 R_{1} and R 2 R_{2} can be reduced to the following form: R ( ψ ) = γ 2 ψ 2 R(\psi)=\frac{\gamma}{2}\psi^{2} (11)
#### Lemma B.4.
The eigenvalues of the Jacobian of the gradient vector field for the gradient-regularized DiracRpGAN at the equilibrium point are given by λ 1 / 2 = − γ 2 ± γ 2 4 − f ′ ( 0 ) \lambda_{1/2}=-\frac{\gamma}{2}\pm\sqrt{\frac{\gamma^{2}}{4}-f^{\prime}(0)} (12) In particular, for γ > 0 \gamma>0 all eigenvalues have a negative real part. Hence, gradient descent is locally convergent for small enough learning rates.
With regularization, the gradient vector field becomes v ~ ( θ , ψ ) = ( − ψ f ′ ( ψ θ ) θ f ′ ( ψ θ ) − γ ψ ) \tilde{v}(\theta,\psi)=\begin{pmatrix}-\psi f^{\prime}(\psi\theta)\\ \theta f^{\prime}(\psi\theta)-\gamma\psi\end{pmatrix} (13) the Jacobian of v ~ \tilde{v} is then given by J v ~ = ( − ψ 2 f ′′ ( ψ θ ) − f ′ ( ψ θ ) − ψ θ f ′′ ( ψ θ ) f ′ ( ψ θ ) + ψ θ f ′′ ( ψ θ ) θ 2 f ′′ ( ψ θ ) − γ ) \textbf{J}_{\tilde{v}}=\begin{pmatrix}-\psi^{2}f^{\prime\prime}(\psi\theta)&-f^{\prime}(\psi\theta)-\psi\theta f^{\prime\prime}(\psi\theta)\\ f^{\prime}(\psi\theta)+\psi\theta f^{\prime\prime}(\psi\theta)&\theta^{2}f^{\prime\prime}(\psi\theta)-\gamma\end{pmatrix} (14) evaluating the Jacobian at θ = ψ = 0 \theta=\psi=0 yields J v ~ | ( 0 , 0 ) = ( 0 − f ′ ( 0 ) f ′ ( 0 ) − γ ) \textbf{J}_{\tilde{v}}\biggr\rvert_{(0,0)}=\begin{pmatrix}0&-f^{\prime}(0)\\ f^{\prime}(0)&-\gamma\end{pmatrix} (15) given some calculations, we arrive at Eq. 12 .
## Appendix C General Convergence Results
#### Summary.
The proofs are largely the same as Mescheder et al . Mescheder et al. [2018] . We use the same proving techniques, and only slightly modify the assumptions and proof details to adapt Mescheder et al . ’s effort to RpGAN. Like in Mescheder et al. [2018] , our proofs do not rely on unrealistic assumptions such as supp p 𝒟 = supp p θ \supp p_{\mathcal{D}}=\supp p_{\theta} .
### C.1 Assumptions
We closely follow Mescheder et al. [2018] but modify the assumptions wherever necessary to tailor the proofs for RpGAN. Like in Mescheder et al. [2018] , we also consider the realizable case where there exists θ \theta such that G θ G_{\theta} produces the true data distribution.
#### Assumption I.
We have p θ ∗ = p 𝒟 p_{\theta^{*}}=p_{\mathcal{D}} , and D ψ ∗ = C D_{\psi^{*}}=C in some local neighborhood of supp p 𝒟 \supp p_{\mathcal{D}} , where C C is some arbitrary constant.
Since RpGAN is defined on critic difference rather than raw logits, we no longer require D ψ ∗ D_{\psi^{*}} to produce 0 on supp p 𝒟 \supp p_{\mathcal{D}} , instead any constant C C would suffice.
#### Assumption II.
We have f ′ ( 0 ) ≠ 0 f^{\prime}(0)\neq 0 and f ′′ ( 0 ) < 0 f^{\prime\prime}(0)<0 .
This assumption is the same as in Mescheder et al. [2018] . The choice f ( t ) = − log ( 1 + e − t ) f(t)=-\log(1+e^{-t}) adopted in the main text satisfies this assumption.
As discussed in Mescheder et al. [2018] , there generally is not a single equilibrium point ( θ ∗ , ψ ∗ ) (\theta^{*},\psi^{*}) , but a submanifold of equivalent equilibria corresponding to different parameterizations of the same function. It is therefore necessary to represent the equilibrium as reparameterization manifolds ℳ G \mathcal{M}_{G} and ℳ D \mathcal{M}_{D} . We modify the reparameterization h h as follows: h ( ψ ) = 𝔼 x ∼ p 𝒟 y ∼ p 𝒟 [ | D ψ ( x ) − D ψ ( y ) | 2 + ‖ ∇ x D ψ ( x ) ‖ 2 ] h(\psi)=\mathbb{E}_{\begin{subarray}{c}x\sim p_{\mathcal{D}}\\ y\sim p_{\mathcal{D}}\end{subarray}}\left[\left|D_{\psi}(x)-D_{\psi}(y)\right|^{2}+\left\|\nabla_{x}D_{\psi}(x)\right\|^{2}\right] (16) to account for the fact that D ψ ∗ D_{\psi^{*}} is now allowed to have any constant value on supp p 𝒟 \supp p_{\mathcal{D}} . The reparameterization manifolds are then given by: ℳ G = { θ | p θ = p 𝒟 } \displaystyle\mathcal{M}_{G}=\{\theta\,\rvert\,p_{\theta}=p_{\mathcal{D}}\} (17) ℳ D = { ψ | h ( ψ ) = 0 } \displaystyle\mathcal{M}_{D}=\{\psi\,\rvert\,h(\psi)=0\} (18) We assume the same regularity properties as in Mescheder et al. [2018] for ℳ G \mathcal{M}_{G} and ℳ D \mathcal{M}_{D} near the equilibrium. To state these assumptions, we need: g ( θ ) = 𝔼 x ∼ p θ [ ∇ ψ D ψ | ψ = ψ ∗ ] g(\theta)=\mathbb{E}_{x\sim p_{\theta}}\left[\nabla_{\psi}D_{\psi}\rvert_{\psi=\psi^{*}}\right] (19) which leads to:
#### Assumption III.
There are ϵ \epsilon -balls B ϵ ( θ ∗ ) B_{\epsilon}(\theta^{*}) and B ϵ ( ψ ∗ ) B_{\epsilon}(\psi^{*}) around θ ∗ \theta^{*} and ψ ∗ \psi^{*} so that ℳ G ∩ B ϵ ( θ ∗ ) \mathcal{M}_{G}\,\cap\,B_{\epsilon}(\theta^{*}) and ℳ D ∩ B ϵ ( ψ ∗ ) \mathcal{M}_{D}\,\cap\,B_{\epsilon}(\psi^{*}) define 𝒞 1 \mathcal{C}^{1} -manifolds. Moreover, the following holds : (i) if v ∈ ℝ n v\in\mathbb{R}^{n} is not in 𝒯 ψ ∗ ℳ D \mathcal{T}_{\psi^{*}}\mathcal{M}_{D} , then ∂ v 2 h ( ψ ∗ ) ≠ 0 \partial_{v}^{2}h(\psi^{*})\neq 0 .
(ii) if w ∈ ℝ m w\in\mathbb{R}^{m} is not in 𝒯 θ ∗ ℳ G \mathcal{T}_{\theta^{*}}\mathcal{M}_{G} , then ∂ w g ( θ ∗ ) ≠ 0 \partial_{w}g(\theta^{*})\neq 0 .
These two conditions have exactly the same meanings as in Mescheder et al. [2018] : the first condition indicates the geometry of ℳ D \mathcal{M}_{D} can be locally described by the second derivative of h h . The second condition implies that D D is strong enough that it can detect any deviation from the equilibrium generator distribution. This is the only assumption we have about the expressiveness of D D .
### C.2 Convergence
We can now show the general convergence result for gradient penalized RpGAN, consider the gradient vector field with either R 1 R_{1} or R 2 R_{2} regularization: v ~ i ( θ , ψ ) = ( − ∇ θ ℒ ( θ , ψ ) ∇ ψ ℒ ( θ , ψ ) − ∇ ψ R i ( θ , ψ ) ) \tilde{v}_{i}(\theta,\psi)=\begin{pmatrix}-\nabla_{\theta}\mathcal{L}(\theta,\psi)\\ \nabla_{\psi}\mathcal{L}(\theta,\psi)-\nabla_{\psi}R_{i}(\theta,\psi)\end{pmatrix} (20) note that the convergence result can also be trivially extended to the case where both R 1 R_{1} and R 2 R_{2} are applied. We omit the proof for this case as it is redundant once the convergence with either regularization is proven.
#### Theorem.
Assume Assumption I, II and III hold for ( θ ∗ , ψ ∗ ) (\theta^{*},\psi^{*}) . For small enough learning rates, gradient descent for v ~ 1 \tilde{v}_{1} and v ~ 2 \tilde{v}_{2} are both convergent to ℳ G × ℳ D \mathcal{M}_{G}\times\mathcal{M}_{D} in a neighborhood of ( θ ∗ , ψ ∗ ) (\theta^{*},\psi^{*}) . Moreover, the rate of convergence is at least linear .
We extend the convergence proof by Mescheder et al . Mescheder et al. [2018] to our setting. We first prove lemmas necessary to our main proof.
#### Lemma C.2.1.
Assume J ∈ ℝ ( n + m ) × ( n + m ) J\in\mathbb{R}^{(n+m)\times(n+m)} is of the following form : J = ( 0 − B ⊤ B − Q ) J=\begin{pmatrix}0&-B^{\top}\\ B&-Q\end{pmatrix} (21) where Q ∈ ℝ m × m Q\in\mathbb{R}^{m\times m} is a symmetric positive definite matrix and B ∈ ℝ m × n B\in\mathbb{R}^{m\times n} has full column rank. Then all eigenvalues λ \lambda of J J satisfy ℜ ( λ ) < 0 \Re(\lambda)<0 .
Proof. See Mescheder et al . Mescheder et al. [2018] , Theorem A.7.
#### Lemma C.2.2.
The gradient of ℒ ( θ , ψ ) \mathcal{L}(\theta,\psi) w.r.t. θ \theta and ψ \psi are given by : ∇ θ ℒ ( θ , ψ ) = 𝔼 z ∼ p z x ∼ p 𝒟 [ f ′ ( D ψ ( G θ ( z ) ) − D ψ ( x ) ) [ ∇ θ G θ ( z ) ] ⊤ ∇ x D ψ ( G θ ( z ) ) ] \displaystyle\nabla_{\theta}\mathcal{L}(\theta,\psi)=\mathbb{E}_{\begin{subarray}{c}z\sim p_{z}\\ x\sim p_{\mathcal{D}}\end{subarray}}[f^{\prime}(D_{\psi}(G_{\theta}(z))-D_{\psi}(x))\left[\nabla_{\theta}G_{\theta}(z)\right]^{\top}\nabla_{x}D_{\psi}(G_{\theta}(z))] (22) ∇ ψ ℒ ( θ , ψ ) = 𝔼 z ∼ p z x ∼ p 𝒟 [ f ′ ( D ψ ( G θ ( z ) ) − D ψ ( x ) ) ( ∇ ψ D ψ ( G θ ( z ) ) − ∇ ψ D ψ ( x ) ) ] \displaystyle\nabla_{\psi}\mathcal{L}(\theta,\psi)=\mathbb{E}_{\begin{subarray}{c}z\sim p_{z}\\ x\sim p_{\mathcal{D}}\end{subarray}}[f^{\prime}(D_{\psi}(G_{\theta}(z))-D_{\psi}(x))(\nabla_{\psi}D_{\psi}(G_{\theta}(z))-\nabla_{\psi}D_{\psi}(x))] (23) Proof. This is just the chain rule.
#### Lemma C.2.3.
Assume that ( θ ∗ , ψ ∗ ) (\theta^{*},\psi^{*}) satisfies Assumption I. The Jacobian of the gradient vector field v ( θ , ψ ) v(\theta,\psi) at ( θ ∗ , ψ ∗ ) (\theta^{*},\psi^{*}) is then J v | ( θ ∗ , ψ ∗ ) = ( 0 − K D G ⊤ K D G K D D ) \textbf{J}_{v}\biggr\rvert_{(\theta^{*},\psi^{*})}=\begin{pmatrix}0&-K^{\top}_{DG}\\ K_{DG}&K_{DD}\end{pmatrix} (24) the terms K D D K_{DD} and K D G K_{DG} are given by K D D = f ′′ ( 0 ) 𝔼 x ∼ p 𝒟 y ∼ p 𝒟 [ ( ∇ ψ D ψ ∗ ( x ) − ∇ ψ D ψ ∗ ( y ) ) ( ∇ ψ D ψ ∗ ( x ) − ∇ ψ D ψ ∗ ( y ) ) ⊤ ] \displaystyle K_{DD}=f^{\prime\prime}(0)\mathbb{E}_{\begin{subarray}{c}x\sim p_{\mathcal{D}}\\ y\sim p_{\mathcal{D}}\end{subarray}}[(\nabla_{\psi}D_{\psi^{*}}(x)-\nabla_{\psi}D_{\psi^{*}}(y))(\nabla_{\psi}D_{\psi^{*}}(x)-\nabla_{\psi}D_{\psi^{*}}(y))^{\top}] (25) K D G = f ′ ( 0 ) ∇ θ 𝔼 x ∼ p θ [ ∇ ψ D ψ ∗ ( x ) ] | θ = θ ∗ \displaystyle K_{DG}=f^{\prime}(0)\nabla_{\theta}\mathbb{E}_{x\sim p_{\theta}}[\nabla_{\psi}D_{\psi^{*}}(x)]\,\rvert_{\theta=\theta^{*}} (26)
Proof. Note that J v | ( θ ∗ , ψ ∗ ) = ( − ∇ θ 2 ℒ ( θ ∗ , ψ ∗ ) − ∇ θ , ψ 2 ℒ ( θ ∗ , ψ ∗ ) ∇ θ , ψ 2 ℒ ( θ ∗ , ψ ∗ ) ∇ ψ 2 ℒ ( θ ∗ , ψ ∗ ) ) \textbf{J}_{v}\biggr\rvert_{(\theta^{*},\psi^{*})}=\begin{pmatrix}-\nabla^{2}_{\theta}\mathcal{L}(\theta^{*},\psi^{*})&-\nabla^{2}_{\theta,\psi}\mathcal{L}(\theta^{*},\psi^{*})\\ \nabla^{2}_{\theta,\psi}\mathcal{L}(\theta^{*},\psi^{*})&\nabla^{2}_{\psi}\mathcal{L}(\theta^{*},\psi^{*})\end{pmatrix} (27) By Assumption I, D ψ ∗ = C D_{\psi^{*}}=C in some neighborhood of supp p 𝒟 \supp p_{\mathcal{D}} . Therefore we also have ∇ x D ψ ∗ = 0 \nabla_{x}D_{\psi^{*}}=0 and ∇ x 2 D ψ ∗ = 0 \nabla^{2}_{x}D_{\psi^{*}}=0 for x ∈ supp p 𝒟 x\in\supp p_{\mathcal{D}} . Using these two conditions, we see that ∇ θ 2 ℒ ( θ ∗ , ψ ∗ ) = 0 \nabla^{2}_{\theta}\mathcal{L}(\theta^{*},\psi^{*})=0 .
To see Eq. 25 and Eq. 26 , simply take the derivatives of Eq. 23 and evaluate at ( θ ∗ , ψ ∗ ) (\theta^{*},\psi^{*}) .
#### Lemma C.2.4.
The gradient ∇ ψ R i ( θ , ψ ) \nabla_{\psi}R_{i}(\theta,\psi) of the regularization terms R i R_{i} , i ∈ { 1 , 2 } i\in\{1,2\} , w.r.t. ψ \psi are ∇ ψ R 1 ( θ , ψ ) = γ 𝔼 x ∼ p 𝒟 [ ∇ ψ , x D ψ ∇ x D ψ ] \displaystyle\nabla_{\psi}R_{1}(\theta,\psi)=\gamma\mathbb{E}_{x\sim p_{\mathcal{D}}}[\nabla_{\psi,x}D_{\psi}\nabla_{x}D_{\psi}] (28) ∇ ψ R 2 ( θ , ψ ) = γ 𝔼 x ∼ p θ [ ∇ ψ , x D ψ ∇ x D ψ ] \displaystyle\nabla_{\psi}R_{2}(\theta,\psi)=\gamma\mathbb{E}_{x\sim p_{\theta}}[\nabla_{\psi,x}D_{\psi}\nabla_{x}D_{\psi}] (29)
Proof. See Mescheder et al . Mescheder et al. [2018] , Lemma D.3.
#### Lemma C.2.5.
The second derivatives ∇ ψ 2 R i ( θ ∗ , ψ ∗ ) \nabla^{2}_{\psi}R_{i}(\theta^{*},\psi^{*}) of the regularization terms R i R_{i} , i ∈ { 1 , 2 } i\in\{1,2\} , w.r.t. ψ \psi at ( θ ∗ , ψ ∗ ) (\theta^{*},\psi^{*}) are both given by L D D = γ 𝔼 x ∼ p 𝒟 [ A A ⊤ ] L_{DD}=\gamma\mathbb{E}_{x\sim p_{\mathcal{D}}}[AA^{\top}] (30) where A = ∇ ψ , x D ψ ∗ A=\nabla_{\psi,x}D_{\psi^{*}} . Moreover, both regularization terms satisfy ∇ θ , ψ R i ( θ ∗ , ψ ∗ ) = 0 \nabla_{\theta,\psi}R_{i}(\theta^{*},\psi^{*})=0 .
Proof. See Mescheder et al . Mescheder et al. [2018] , Lemma D.4.
Given Lemma C.2.3, Lemma C.2.5 and Eq. 20 , we can now show that the Jacobian of the regularized gradient field at the equilibrium point is given by J v ~ | ( θ ∗ , ψ ∗ ) = ( 0 − K D G ⊤ K D G M D D ) \textbf{J}_{\tilde{v}}\biggr|_{(\theta^{*},\psi^{*})}=\begin{pmatrix}0&-K_{DG}^{\top}\\ K_{DG}&M_{DD}\end{pmatrix} (31) where M D D = K D D − L D D M_{DD}=K_{DD}-L_{DD} . To prove our main theorem, we need to examine J v ~ \textbf{J}_{\tilde{v}} when restricting it to the space orthogonal to 𝒯 ( θ ∗ , ψ ∗ ) ℳ G × ℳ D \mathcal{T}_{(\theta^{*},\psi^{*})}\mathcal{M}_{G}\times\mathcal{M}_{D} .
#### Lemma C.2.6.
Assume Assumptions II and III hold. If v ≠ 0 v\neq 0 is not in 𝒯 ψ ∗ ℳ D \mathcal{T}_{\psi^{*}}\mathcal{M}_{D} , then v ⊤ M D D v < 0 v^{\top}M_{DD}v<0 .
Proof. By Lemma C.2.3 and Lemma C.2.5, we have v ⊤ K D D v = f ′′ ( 0 ) 𝔼 x ∼ p 𝒟 y ∼ p 𝒟 [ ( ( ∇ ψ D ψ ∗ ( x ) − ∇ ψ D ψ ∗ ( y ) ) ⊤ v ) 2 ] \displaystyle v^{\top}K_{DD}v=f^{\prime\prime}(0)\mathbb{E}_{\begin{subarray}{c}x\sim p_{\mathcal{D}}\\ y\sim p_{\mathcal{D}}\end{subarray}}\left[((\nabla_{\psi}D_{\psi^{*}}(x)-\nabla_{\psi}D_{\psi^{*}}(y))^{\top}v)^{2}\right] (32) v ⊤ L D D v = γ 𝔼 x ∼ p 𝒟 [ ‖ A v ‖ 2 ] \displaystyle v^{\top}L_{DD}v=\gamma\mathbb{E}_{x\sim p_{\mathcal{D}}}\left[\left\|Av\right\|^{2}\right] (33) By Assumption II, we have f ′′ ( 0 ) < 0 f^{\prime\prime}(0)<0 . Therefore v ⊤ M D D v ≤ 0 v^{\top}M_{DD}v\leq 0 . Suppose v ⊤ M D D v = 0 v^{\top}M_{DD}v=0 , this implies ( ∇ ψ D ψ ∗ ( x ) − ∇ ψ D ψ ∗ ( y ) ) ⊤ v = 0 and A v = 0 (\nabla_{\psi}D_{\psi^{*}}(x)-\nabla_{\psi}D_{\psi^{*}}(y))^{\top}v=0\,\,\,\,\text{and}\,\,\,\,Av=0 (34) for all ( x , y ) ∈ supp p 𝒟 × supp p 𝒟 (x,y)\in\supp p_{\mathcal{D}}\times\supp p_{\mathcal{D}} . Recall the definition of h ( ψ ) h(\psi) from Eq. 16 . Using the fact that D ψ ∗ = C D_{\psi^{*}}=C and ∇ x D ψ ∗ = 0 \nabla_{x}D_{\psi^{*}}=0 for x ∈ supp p 𝒟 x\in\supp p_{\mathcal{D}} , we see that the Hessian of h ( ψ ) h(\psi) at ψ ∗ \psi^{*} is ∇ ψ 2 h ( ψ ∗ ) = 2 𝔼 x ∼ p 𝒟 y ∼ p 𝒟 [ ( ∇ ψ D ψ ∗ ( x ) − ∇ ψ D ψ ∗ ( y ) ) ( ∇ ψ D ψ ∗ ( x ) − ∇ ψ D ψ ∗ ( y ) ) ⊤ + A A ⊤ ] \nabla^{2}_{\psi}h(\psi^{*})=2\mathbb{E}_{\begin{subarray}{c}x\sim p_{\mathcal{D}}\\ y\sim p_{\mathcal{D}}\end{subarray}}[(\nabla_{\psi}D_{\psi^{*}}(x)-\nabla_{\psi}D_{\psi^{*}}(y))(\nabla_{\psi}D_{\psi^{*}}(x)-\nabla_{\psi}D_{\psi^{*}}(y))^{\top}+AA^{\top}] (35) The second directional derivative ∂ v 2 h ( ψ ) \partial^{2}_{v}h(\psi) is therefore ∂ v 2 h ( ψ ) = 2 𝔼 x ∼ p 𝒟 y ∼ p 𝒟 [ | ( ∇ ψ D ψ ∗ ( x ) − ∇ ψ D ψ ∗ ( y ) ) ⊤ v | 2 + ‖ A v ‖ 2 ] = 0 \partial^{2}_{v}h(\psi)=2\mathbb{E}_{\begin{subarray}{c}x\sim p_{\mathcal{D}}\\ y\sim p_{\mathcal{D}}\end{subarray}}\left[\left|(\nabla_{\psi}D_{\psi^{*}}(x)-\nabla_{\psi}D_{\psi^{*}}(y))^{\top}v\right|^{2}+\left\|Av\right\|^{2}\right]=0 (36) By Assumption III, this can only hold if v ∈ 𝒯 ψ ∗ ℳ D v\in\mathcal{T}_{\psi^{*}}\mathcal{M}_{D} .
#### Lemma C.2.7.
Assume Assumption III holds. If w ≠ 0 w\neq 0 is not in 𝒯 θ ∗ ℳ G \mathcal{T}_{\theta^{*}}\mathcal{M}_{G} , then K D G w ≠ 0 K_{DG}w\neq 0 .
Proof. See Mescheder et al . Mescheder et al. [2018] , Lemma D.6.