Latent Consistency Models: Synthesizing High-Resolution Images with Few-Step Inference — part 4/4
Cleaned-text code points [46964, 62723). Bibliography omitted here; source.md includes it.
In this section, we provide further details of Latent Consistency Fine-tuning (LCF). The pseudo-code of LCF is provided in Algorithm 4 . During the Latent Consistency Fine-tuning (LCF) process, we randomly select two time steps t n t_{n} and t n + k t_{n+k} that are k k time steps apart and apply the same Gaussian noise ϵ \bm{\epsilon} to obtain the noised data 𝒛 t n , 𝒛 t n + k {\bm{z}}_{t_{n}},{\bm{z}}_{t_{n+k}} as follows: 𝒛 t n + k = α ( t n + k ) 𝒛 + σ ( t n + k ) ϵ , 𝒛 t n = α ( t n ) 𝒛 + σ ( t n ) ϵ . {\bm{z}}_{t_{n+k}}=\alpha(t_{n+k}){\bm{z}}+\sigma(t_{n+k})\bm{\epsilon}\quad,\quad{\bm{z}}_{t_{n}}=\alpha(t_{n}){\bm{z}}+\sigma(t_{n})\bm{\epsilon}. Then, we can directly calculate the consistency loss for these two time steps to enforce self-consistency property in Eq. 4 . Notably, this method can also utilize the skipping-step technique to speedup the convergence. Furthermore, we note that latent consistency fine-tuning is independent of the pre-trained teacher model, facilitating direct fine-tuning of a pre-trained latent consistency model without reliance on the teacher diffusion model.
## Appendix D Different ways to parameterize the consistency function
As previously discussed in Eq 9 , we can parameterize our consistency model function 𝒇 𝜽 ( 𝒛 , 𝒄 , t ) {\bm{f}}_{\bm{\theta}}({\bm{z}},{\bm{c}},t) in different ways, depending on the way the teacher diffusion model is parameterized. For ϵ -Prediction \bm{\epsilon}\text{-Prediction} ( Song et al., 2020a ) , we use the following parameterization: 𝒇 𝜽 ( 𝒛 , 𝒄 , t ) = c skip ( t ) 𝒛 + c out ( t ) 𝒛 ^ 0 ( ϵ -Prediction ) {\bm{f}}_{\bm{\theta}}({\bm{z}},{\bm{c}},t)=c_{\text{skip}}(t){\bm{z}}+c_{\text{out}}(t)\hat{{\bm{z}}}_{0}\quad(\bm{\epsilon}\text{-Prediction}) (27) where 𝒛 ^ 0 = ( 𝒛 t − σ ( t ) ϵ ^ θ ( 𝒛 , 𝒄 , t ) α ( t ) ) . \hat{{\bm{z}}}_{0}=\left(\frac{{\bm{z}}_{t}-\sigma(t)\hat{\bm{\epsilon}}_{\theta}({\bm{z}},{\bm{c}},t)}{\alpha(t)}\right). (28) Recalling that 𝒛 t = α ( t ) 𝒛 0 + σ ( t ) ϵ {\bm{z}}_{t}=\alpha(t){\bm{z}}_{0}+\sigma(t)\bm{\epsilon} , 𝒛 ^ 0 \hat{{\bm{z}}}_{0} can be seen as a prediction of 𝒛 0 {\bm{z}}_{0} at time t t .
Next, we provide the parameterization of ( 𝒙 -Prediction ) (\bm{x}\text{-Prediction}) ( Ho et al., 2020 ; Salimans & Ho, 2022 ) with the following form: 𝒇 𝜽 ( 𝒛 , 𝒄 , t ) = c skip ( t ) 𝒛 + c out ( t ) 𝒙 θ ( 𝒛 t , 𝒄 , t ) , ( 𝒙 -Prediction ) {\bm{f}}_{\bm{\theta}}({\bm{z}},{\bm{c}},t)=c_{\text{skip}}(t){\bm{z}}+c_{\text{out}}(t)\bm{x}_{\theta}({\bm{z}}_{t},{\bm{c}},t),\quad(\bm{x}\text{-Prediction}) (29) where 𝒙 θ ( 𝒛 t , 𝒄 , t ) \bm{x}_{\theta}({\bm{z}}_{t},{\bm{c}},t) corresponds to the teacher diffusion model with 𝒙 \bm{x} -prediction.
Finally, for 𝒗 \bm{v} -prediction ( Salimans & Ho, 2022 ) , the consistency function is parameterized as 𝒇 𝜽 ( 𝒛 , 𝒄 , t ) = c skip ( t ) 𝒛 + c out ( t ) ( α t 𝒛 t − σ t 𝒗 θ ( 𝒛 t , 𝒄 , t ) ) , ( 𝒗 -Prediction ) {\bm{f}}_{\bm{\theta}}({\bm{z}},{\bm{c}},t)=c_{\text{skip}}(t){\bm{z}}+c_{\text{out}}(t)\left(\alpha_{t}{\bm{z}}_{t}-\sigma_{t}\bm{v}_{\theta}({\bm{z}}_{t},{\bm{c}},t)\right),\quad(\bm{v}\text{-Prediction}) (30) where 𝒗 θ ( 𝒛 t , 𝒄 , t ) \bm{v}_{\theta}({\bm{z}}_{t},{\bm{c}},t) corresponds to the teacher diffusion model with 𝒗 \bm{v} -prediction.
As mentioned in Sec 5.1 , we use the ϵ -Parameterization \bm{\epsilon}\text{-Parameterization} in Eq. 27 to train LCM at 512 × \times 512 resolution using the teacher diffusion model, Stable-Diffusion-V2.1-Base (originally trained with ϵ -Prediction \bm{\epsilon}\text{-Prediction} at 512 resolution). For resolution 768 × \times 768, we train the LCM using the 𝒗 -Parameterization \bm{v}\text{-Parameterization} in Eq. 30 , adopting the teacher diffusion model, Stable-Diffusion-V2.1 (originally trained with 𝒗 -Prediction \bm{v}\text{-Prediction} at 768 resolution).
## Appendix E Formulas of Other ODE Solvers
As discussed in Sec 4.3 , we use the DDIM ( Song et al., 2020a ) , DPM-Solver ( Lu et al., 2022a ) and DPM-Solver++ ( Lu et al., 2022b ) as the PF-ODE solvers. Proven in ( Lu et al., 2022a ) , the DDIM-Solver is actually the first-order discretization approximation of the DPM-Solver.
For DDIM ( Song et al., 2020a ) , the detailed formula of DDIM PF-ODE solver Ψ DDIM \Psi_{\text{DDIM}} from t n + k t_{n+k} to t n t_{n} is provided as follows. Ψ DDIM ( 𝒛 t n + k , t n + k , t n , 𝒄 ) \displaystyle\Psi_{\text{DDIM}}({\bm{z}}_{t_{n+k}},t_{n+k},t_{n},{\bm{c}}) = 𝒛 ^ t n − 𝒛 t n + k \displaystyle=\hat{{\bm{z}}}_{t_{n}}-{\bm{z}}_{t_{n+k}} (31) = α t n α t n + k 𝒛 t n + k − σ t n ( σ t n + k ⋅ α t n α t n + k ⋅ σ t n − 1 ) ϵ ^ θ ( 𝒛 t n + k , 𝒄 , t n + k ) ⏟ DDIM Estimated 𝒛 t n − 𝒛 t n + k \displaystyle=\underbrace{\frac{\alpha_{t_{n}}}{\alpha_{t_{n+k}}}{\bm{z}}_{t_{n+k}}-\sigma_{t_{n}}\left(\frac{\sigma_{t_{n+k}}\cdot\alpha_{t_{n}}}{\alpha_{t_{n+k}}\cdot\sigma_{t_{n}}}-1\right)\hat{\bm{\epsilon}}_{\theta}({\bm{z}}_{t_{n+k}},{\bm{c}},t_{n+k})}_{\text{DDIM Estimated}\ {\bm{z}}_{t_{n}}}-{\bm{z}}_{t_{n+k}}
For DPM-Solver ( Lu et al., 2022a ) , we only consider the case for o r d e r = 2 order=2 , and the detailed formula of PF-ODE solver Ψ DPM-Solver \Psi_{\text{DPM-Solver}} is provided as follows. First we define some notations. We denote λ t n = log ( α t n σ t n ) \lambda_{t_{n}}=\log(\frac{\alpha_{t_{n}}}{\sigma_{t_{n}}}) , which is the Log-SNR, h t n 0 = λ t n − λ t n + k , h t n 1 = λ t n − λ t n + k / 2 h_{t_{n}}^{0}=\lambda_{t_{n}}-\lambda_{t_{n+k}},h_{t_{n}}^{1}=\lambda_{t_{n}}-\lambda_{t_{n+k/2}} , and r t n = h t n 1 / h t n 0 r_{t_{n}}=h_{t_{n}}^{1}/h_{t_{n}}^{0} . Ψ DPM-Solver ( 𝒛 t n + k , t n + k , t n , 𝒄 ) \displaystyle\Psi_{\text{DPM-Solver}}({\bm{z}}_{t_{n+k}},t_{n+k},t_{n},{\bm{c}}) (32) = α t n α t n + k 𝒛 t n + k − σ t n ( e h t n 0 − 1 ) ϵ ^ θ ( 𝒛 t n + k , 𝒄 , t n + k ) \displaystyle=\frac{\alpha_{t_{n}}}{\alpha_{t_{n+k}}}{\bm{z}}_{t_{n+k}}-\sigma_{t_{n}}(e^{h_{t_{n}}^{0}}-1)\hat{\bm{\epsilon}}_{\theta}({\bm{z}}_{t_{n+k}},{\bm{c}},t_{n+k}) − σ t n 2 r t n ( e h t n 0 − 1 ) ( ϵ ^ θ ( 𝒛 t n + k / 2 Ψ , 𝒄 , t n + k / 2 ) − ϵ ^ θ ( 𝒛 t n + k , 𝒄 , t n + k ) ) − 𝒛 t n + k , \displaystyle-\frac{\sigma_{t_{n}}}{2r_{t_{n}}}(e^{h_{t_{n}}^{0}}-1)\left(\hat{\bm{\epsilon}}_{\theta}({\bm{z}}^{\Psi}_{t_{n+k/2}},{\bm{c}},t_{n+k/2})-\hat{\bm{\epsilon}}_{\theta}({\bm{z}}_{t_{n+k}},{\bm{c}},t_{n+k})\right)-{\bm{z}}_{t_{n+k}}, where ϵ ^ \hat{\bm{\epsilon}} is the noise prediction model, and 𝒛 t n + k / 2 Ψ {\bm{z}}^{\Psi}_{t_{n+k/2}} is the middle point between n + k n+k and n n , given by the following formula: 𝒛 t n + k / 2 Ψ \displaystyle{\bm{z}}^{\Psi}_{t_{n+k/2}} = α t n + k / 2 α t n + k 𝒛 t n + k − σ t n + k / 2 ( e h t n 1 − 1 ) ϵ ^ θ ( 𝒛 t n + k , 𝒄 , t n + k ) \displaystyle=\frac{\alpha_{t_{n+k/2}}}{\alpha_{t_{n+k}}}{\bm{z}}_{t_{n+k}}-\sigma_{t_{n+k/2}}(e^{h_{t_{n}}^{1}}-1)\hat{\bm{\epsilon}}_{\theta}({\bm{z}}_{t_{n+k}},{\bm{c}},t_{n+k}) (33)
For DPM-Solver++ ( Lu et al., 2022b ) , we consider the case for o r d e r = 2 order=2 , DPM-Solver++ replaces the original noise prediction to data prediction ( Lu et al., 2022b ) , with the detailed formula of Ψ DPM-Solver++ \Psi_{\text{DPM-Solver++}} provided as follows. Ψ DPM-Solver++ ( 𝒛 t n + k , t n + k , t n , 𝒄 ) \displaystyle\Psi_{\text{DPM-Solver++}}({\bm{z}}_{t_{n+k}},t_{n+k},t_{n},{\bm{c}}) (34) = σ t n σ t n + k 𝒛 t n + k − α t n ( e − h t n 0 − 1 ) 𝒙 ^ θ ( 𝒛 t n + k , 𝒄 , t n + k ) \displaystyle=\frac{\sigma_{t_{n}}}{\sigma_{t_{n+k}}}{\bm{z}}_{t_{n+k}}-\alpha_{t_{n}}(e^{-h_{t_{n}}^{0}}-1)\hat{\bm{x}}_{\theta}({\bm{z}}_{t_{n+k}},{\bm{c}},t_{n+k}) − α t n 2 r t n ( e − h t n 0 − 1 ) ( 𝒙 ^ θ ( 𝒛 t n + k / 2 Ψ , 𝒄 , t n + k / 2 ) − 𝒙 ^ θ ( 𝒛 t n + k , 𝒄 , t n + k ) ) − 𝒛 t n + k , \displaystyle-\frac{\alpha_{t_{n}}}{2r_{t_{n}}}(e^{-h_{t_{n}}^{0}}-1)\left(\hat{\bm{x}}_{\theta}({\bm{z}}^{\Psi}_{t_{n+k/2}},{\bm{c}},t_{n+k/2})-\hat{\bm{x}}_{\theta}({\bm{z}}_{t_{n+k}},{\bm{c}},t_{n+k})\right)-{\bm{z}}_{t_{n+k}}, where 𝒙 ^ \hat{\bm{x}} is the data prediction model ( Lu et al., 2022a ) and 𝒛 t n + k / 2 Ψ {\bm{z}}^{\Psi}_{t_{n+k/2}} is the middle point between n + k n+k and n n , given by the following formula: 𝒛 t n + k / 2 Ψ \displaystyle{\bm{z}}^{\Psi}_{t_{n+k/2}} = σ t n + k / 2 σ t n + k 𝒛 t n + k − α t n + k / 2 ( e − h t n 1 − 1 ) 𝒙 ^ θ ( 𝒛 t n + k , 𝒄 , t n + k ) \displaystyle=\frac{\sigma_{t_{n+k/2}}}{\sigma_{t_{n+k}}}{\bm{z}}_{t_{n+k}}-\alpha_{t_{n+k/2}}(e^{-h_{t_{n}}^{1}}-1)\hat{\bm{x}}_{\theta}({\bm{z}}_{t_{n+k}},{\bm{c}},t_{n+k}) (35)
## Appendix F Training Details of Latent Consistency Distillation
As mentioned in Section 5.1 , we conduct our experiments in two resolution settings 512 × \times 512 and 768 × \times 768. For the former setting, we use the LAION-Aesthetics-6+ ( Schuhmann et al., 2022 ) 12M dataset, consisting of 12M text-image pairs with predicted aesthetics scores higher than 6. For the latter setting, we use the LAIOIN-Aesthetic-6.5+ ( Schuhmann et al., 2022 ) , which comprise 650K text-image pairs with predicted aesthetics scores higher than 6.5.
For 512 × \times 512 resolution, we train the LCM with the teacher diffusion model Stable-Diffusion-V2.1-Base (SD-V2.1-Base) ( Rombach et al., 2022 ) , which is originally trained on 512 × \times 512 resolution images using the ϵ \bm{\epsilon} -Prediction ( Ho et al., 2020 ) . We train LCM (512 × \times 512) with 100K iterations on 8 A100 GPUs, using a batch size of 72, the same learning rate 8e-6 , EMA rate μ = 0.999943 \mu=0.999943 and Rectified Adam optimizer ( Liu et al., 2019 ) used in ( Song et al., 2023 ) . We select the DDIM-Solver ( Song et al., 2020a ) and skipping step k = 20 k=20 in Eq. 17 . We set the guidance scale range [ ω min , ω max ] = [ 2 , 14 ] [\omega_{\text{min}},\omega_{\text{max}}]=[2,14] , which is consistent with the setting in Guided-Distill ( Meng et al., 2023 ) . During training, we initialize the consistency function 𝒇 𝜽 ( 𝒛 t n , ω , 𝒄 , t n ) \bm{f_{\theta}}(\bm{{\bm{z}}}_{t_{n}},\omega,\bm{c},t_{n}) with the same parameters as the teacher diffusion model (SD-V2.1-Base). To encode the CFG scale ω \omega into the LCM, we applying Fourier embedding to ω \omega , integrating it into the origin LCM backbone by adding the projected ω \omega -embedding into the original embedding, as done in ( Meng et al., 2023 ) . We use a zero parameter initialization method mentioned in ( Zhang & Agrawala, 2023 ) on projected ω \omega -embedding for better training stability. For training LCM (512 × \times 512), we use a augmented consistency function parameterized in ϵ \bm{\epsilon} -prediction as discussed in Appendix. D .
For 768 × \times 768 resolution, we train the LCM with the teacher diffusion model Stable-Diffusion-V2.1 (SD-V2.1) ( Rombach et al., 2022 ) , which is originally trained on 768 × \times 768 resolution images using the 𝒗 \bm{v} -Prediction ( Salimans & Ho, 2022 ) . We train LCM (768 × \times 768) with 100K iterations on 8 A100 GPUs using a batch size of 16, while the other hyper-parameters remain the same as in 512 × \times 512 resolution setting.
## Appendix G Reproduction Details of Guided-Distill
Guided-Distill ( Meng et al., 2023 ) serves as a significant baseline for guided distillation but is not open-sourced. We adhered strictly to the training procedure described in the paper, reproducing the method for accurate comparisons. For 512 × \times 512 resolution setting, Guided-Distill ( Meng et al., 2023 ) used a large batch size of 512, which requires at least 32 A100 GPUs for training. Due to limited resource, we reduced the batch size to 72 (512 resolution), while set the batchsize to 16 for 768 resolution, the same as ours, and trained for 100K iterations, also the same as in LCM.
Specifically, Guided Distill involves two stages of distillation. For the first stage, it use a student model to fit the outputs of the pre-trained guided diffusion model using classifier-free guidance scales ω \omega . The loss function is as follows: 𝔼 w ∼ p w , t ∼ 𝒰 [ 0 , 1 ] , 𝒙 ∼ p data ( 𝒙 ) [ ω ( λ t ) ‖ 𝒙 ^ 𝜼 1 ( 𝒛 t , w ) − 𝒙 ^ 𝜽 w ( 𝒛 t ) ‖ 2 2 ] , \mathbb{E}_{w\sim p_{w},t\sim\mathcal{U}[0,1],{\bm{x}}\sim p_{\text{data}}}({\bm{x}})[\omega(\lambda_{t})||\hat{{\bm{x}}}_{\bm{\eta}_{1}}({\bm{z}}_{t},w)-\hat{{\bm{x}}}_{\bm{\theta}}^{w}({\bm{z}}_{t})||_{2}^{2}], (36) where 𝒙 ^ 𝜽 ( 𝒛 t ) = ( 1 + w ) 𝒙 ^ c , 𝜽 ( 𝒛 t ) − w 𝒙 ^ 𝜽 ( 𝒛 t ) , 𝒛 t ∼ q ( 𝒛 t | 𝒙 ) \hat{{\bm{x}}}_{\bm{\theta}}({\bm{z}}_{t})=(1+w)\hat{{\bm{x}}}_{c,{\bm{\theta}}}({\bm{z}}_{t})-w\hat{{\bm{x}}}_{\bm{\theta}}({\bm{z}}_{t}),{\bm{z}}_{t}\sim q({\bm{z}}_{t}|{\bm{x}}) and p w ( w ) = 𝒰 [ w min , w max ] p_{w}(w)=\mathcal{U}[w_{\text{min}},w_{\text{max}}] .
In our implementation, we follow the same training procedure in ( Meng et al., 2023 ) except the difference of computation resources. For first stage distillation , we train the student model with 25,000 gradient updates (batch size 72), roughly the same computation costs in ( Meng et al., 2023 ) (3,000 gradient updates, batch size 512), and we reduce the original learning rate 1 e − 4 1e-4 to 5 e − 5 5e-5 for smaller batch size. For second stage distillation , we progressively train the student model using the same schedule as in Guided-Distill ( Meng et al., 2023 ) except for batch size difference. We train the student model with 2500 gradient updates except when the sampling step equals to 1,2, or 4, where we train for 20000 gradient updates, the same schedule used in ( Meng et al., 2023 ) . We trained until the total number of gradient iterations for the entire stage reached 100K the same as LCM training. The generation results of Guided Distill are shown in Figure 2 . We can also see that the performances in Table 1 and Table 2 are similar, further verifying the correctness of our Guided-Distill implementation. Nevertheless, we acknowledge that longer training and more computational resources can lead to better results as reported in ( Meng et al., 2023 ) . However, LCM achieves faster convergence and superior results under the same computation cost (same batch size, same number of iterations), demonstrating its practicability and superiority.
## Appendix H More few-step Inference Results
We present more images (768 × \times 768) generation results with LCM using 4 and 2-steps inference in Figure 7 and Figure 8 . It is evident that LCM is capable of synthesizing high-resolution images with just 2, 4 steps of inference. Moreover, LCM can be derived from any pre-trained Stable Diffusion (SD) ( Rombach et al., 2022 ) in merely 4,000 training steps, equivalent to around 32 A100 GPU Hours, showcasing the effectiveness and superiority of LCM.
## Instructions for reporting errors
We are continuing to improve HTML versions of papers, and your feedback helps enhance accessibility and mobile support. To report errors in the HTML that will help us improve conversion and rendering, choose any of the methods listed below:
Click the "Report Issue" ( ) button, located in the page header.
Tip: You can select the relevant text first, to include it in your report.
Our team has already identified the following issues . We appreciate your time reviewing and reporting rendering errors we may not have found yet. Your efforts will help us improve the HTML versions for all readers, because disability should not be a barrier to accessing research. Thank you for your continued support in championing open access for all.
Have a free development cycle? Help support accessibility at arXiv! Our collaborators at LaTeXML maintain a list of packages that need conversion , and welcome developer contributions .