Latent Consistency Models: Synthesizing High-Resolution Images with Few-Step Inference — part 3/4

Cleaned-text code points [31306, 46964). Bibliography omitted here; source.md includes it.

In this section, we employ latency consistency distillation to train LCM on two subsets of LAION-5B. In Sec 5.1 , we first evaluate the performance of LCM on text-to-image generation tasks. In Sec 5.2 , we provide a detailed ablation study to test the effectiveness of using different solvers, skipping step schedules and guidance scales. Lastly, in Sec 5.3 , we present the experimental results of latent consistency finetuning on a pretrained LCM on customized image datasets.

### 5.1 Text-to-Image Generation

Datasets We use two subsets of LAION-5B ( Schuhmann et al., 2022 ) : LAION-Aesthetics-6+ (12M) and LAION-Aesthetics-6.5+ (650K) for text-to-image generation. Our experiments consider resolutions of 512 × \times 512 and 768 × \times 768. For 512 resolution, we use LAION-Aesthetics-6+, which comprises 12M text-image pairs with predicted aesthetics scores higher than 6. For 768 resolution, we use LAION-Aesthetics-6.5+, with 650K text-image pairs with aesthetics score higher than 6.5.

Model Configuration For 512 resolution, we use the pre-trained Stable Diffusion-V2.1-Base ( Rombach et al., 2022 ) as the teacher model, which was originally trained on resolution 512 × \times 512 with ϵ \bm{\epsilon} -Prediction ( Ho et al., 2020 ) . For 768 resolution, we use the widely used pre-trained Stable Diffusion-V2.1, originally trained on resolution 768 × \times 768 with 𝒗 \bm{v} -Prediction ( Salimans & Ho, 2022 ) . We train LCM with 100K iterations and we use a batch size of 72 for ( 512 × 512 ) (512\times 512) setting, and 16 for ( 768 × 768 ) (768\times 768) setting, the same learning rate 8e-6 and EMA rate μ = 0.999943 \mu=0.999943 as used in ( Song et al., 2023 ) . For augmented PF-ODE solver Ψ \Psi and skipping step k k in Eq. 17 , we use DDIM-Solver ( Song et al., 2020a ) with skipping step k = 20 k=20 . We set the guidance scale range [ w min , w max ] = [ 2 , 14 ] [w_{\text{min}},w_{\text{max}}]=[2,14] , consistent with ( Meng et al., 2023 ) . More training details are provided in the Appendix F .

Baselines & Evaluation We use DDIM ( Song et al., 2020a ) , DPM ( Lu et al., 2022a ) , DPM++ ( Lu et al., 2022b ) and Guided-Distill ( Meng et al., 2023 ) as baselines. The first three are training-free samplers requiring more peak memory per step with classifier-free guidance. Guided-Distill requires two stages of guided distillation. Since Guided-Distill is not open-sourced, we strictly followed the training procedure outlined in the paper to reproduce the results. Due to the limited resource ( Meng et al. (2023) used a large batch size of 512, requiring at least 32 A100 GPUs), we reduce the batch size to 72 72 , the same as ours, and trained for the same 100K iterations. Reproduction details are provided in Appendix G . We admit that longer training and more computational resources can lead to better results as reported in ( Meng et al., 2023 ) . However, LCM achieves faster convergence and superior results under the same computation cost. For evaluation, We generate 30K images from 10K text prompts in the test set (3 images per prompt), and adopt FID and CLIP scores to evaluate the diversity and quality of the generated images. We use ViT-g/14 for evaluating CLIP scores.

Results. The quantitative results in Tables 1 and 2 show that LCM notably outperforms baseline methods at 512 512 and 768 768 resolutions, especially in the low step regime (1 ∼ \sim 4), highlighting its efficency and superior performance. Unlike DDIM, DPM, DPM++, which require more peak memory per sampling step with CFG, LCM requires only one forward pass per sampling step, saving both time and memory. Moreover, in contrast to the two-stage distillation procedure employed in Guided-Distill, LCM only needs one-stage guided distillation, which is much simpler and more practical. The qualitative results in Figure 2 further show the superiority of LCM with 2- and 4-step inference.

### 5.2 Ablation Study

ODE Solvers & Skipping-Step Schedule. We compare various solvers Ψ \Psi (DDIM ( Song et al., 2020a ) , DPM ( Lu et al., 2022a ) , DPM++ ( Lu et al., 2022b ) ) for solving the augmented PF-ODE specified in Eq 17 , and explore different skipping step schedules with different k k . The results are depicted in Figure 3 . We observe that: 1) Using Skipping-Step techniques (see Sec 4.3 ), LCM achieves fast convergence within 2,000 iterations in the 4-step inference setting. Specifically, the DDIM solver converges slowly at skipping step k = 1 k=1 , while setting k = 5 , 10 , 20 k=5,10,20 leads to much faster convergence, underscoring the effectiveness of the Skipping-Step method. 2) DPM and DPM++ solvers perform better at a larger skipping step ( k = 50 k=50 ) compared to the DDIM solver which suffers from increased ODE approximation error with larger k k . This phenomenon is also discussed in ( Lu et al., 2022a ) . 3) Very small k k values (1 or 5) result in slow convergence and very large ones (e.g., 50 for DDIM) may lead to inferior results. Hence, we choose k = 20 k=20 , which provides competitive performance for all three solvers, for our main experiment in Sec 5.1 .

The Effect of Guidance Scale ω \omega . We examine the effect of using different CFG scales ω \omega in LCM. Typically, ω \omega balances sample quality and diversity. A larger ω \omega generally tends to improve sample quality (indicated by CLIP), but may compromise diversity (measured by FID). Beyond a certain threshold, an increased ω \omega yields better CLIP scores at the expense of FID. Figure 4 presents the results for various ω \omega across different inference steps. Our findings include: 1) Using large ω \omega enhances sample quality (CLIP Scores) but results in relatively inferior FID. 2) The performance gaps across 2, 4, and 8 inference steps are negligible, highlighting LCM’s efficacy in 2 ∼ \sim 8 step regions. However, a noticeable gap exists in one-step inference, indicating rooms for further improvements. We present visualizations for different ω \omega in Figure 5 . One can see clearly that a larger ω \omega enhances sample quality, verifying the effectiveness of our one-stage guided distillation method.

### 5.3 Downstream Consistency Fine-tuning Results

We perform Latent Consistency Fine-tuning (LCF) on two customized image datasets, Pokemon dataset ( Pinkney, 2022 ) and Simpsons dataset ( Norod78, 2022 ) , to demonstrate the efficiency of LCF. Each dataset, comprised of hundreds of customized text-image pairs, is split such that 90% is used for fine-tuning and the rest 10% for testing. For LCF, we utilize pretrained LCM that was originally trained at the resolution of 768 × \times 768 used in Table 2 . For these two datasets, we fine-tune the pre-trained LCM for 30K iterations with a learning rate 8e-6. We present qualitative results of adopting LCF on two customized image datasets in Figure 6 . The finetuned LCM is capable of generating images with customized styles in few steps, showing the effectiveness of our method.

## 6 Conclusion

We present Latent Consistency Models (LCMs), and a highly efficient one-stage guided distillation method that enables few-step or even one-step inference on pre-trained LDMs. Furthermore, we present latent consistency fine-tuning (LCF), to enable few-step inference of LCMs on customized image datasets. Extensive experiments on the LAION-5B-Aesthetics dataset demonstrate the superior performance and efficiency of LCMs. Future work include extending our method to more image generation tasks such as text-guided image editing, inpainting and super-resolution.

## Appendix A More Details on Diffusion and Consistency Models

### A.1 Diffusion Models

Consider the forward process, described by the following SDE for t ∈ [ 0 , T ] t\in[0,T] : d ​ 𝒙 t = f ⁡ ( t ) ​ 𝒙 t ​ d ​ t + g ⁡ ( t ) ​ d ​ 𝒘 t , 𝒙 0 ∼ p d ​ a ​ t ​ a ​ ( 𝒙 0 ) , \mathrm{d}\bm{x}_{t}=f(t)\bm{x}_{t}\mathrm{~d}t+g(t)\mathrm{d}\bm{w}_{t},\quad\bm{x}_{0}\sim p_{data}\left(\bm{x}_{0}\right), (19) where 𝒘 t \bm{w}_{t} denotes the standard Brownian motion. Leveraging the classic result of Anderson (1982), Song et al. (2020b) show that the reverse process of the above forward process is also a diffusion process, specified by the following reverse-time SDE: d ​ 𝒙 t = [ f ⁡ ( t ) ​ 𝒙 t − g 2 ​ ( t ) ​ ∇ 𝒙 ​ log ⁡ q t ​ ( 𝒙 t ) ] ​ d ​ t + g ⁡ ( t ) ​ d ​ 𝒘 ¯ t , 𝒙 T ∼ q T ​ ( 𝒙 T ) , \mathrm{d}\bm{x}_{t}=\left[f(t)\bm{x}_{t}-g^{2}(t)\nabla_{\bm{x}}\log q_{t}\left(\bm{x}_{t}\right)\right]\mathrm{d}t+g(t)\mathrm{d}\overline{\bm{w}}_{t},\quad\bm{x}_{T}\sim q_{T}\left(\bm{x}_{T}\right), (20) where 𝒘 ¯ t \overline{\bm{w}}_{t} is a standard reverse-time Brownian motion. One can leverage the reverse SDE for data sampling from T T to 0 0 , starting with q T ​ ( 𝒙 T ) q_{T}(\bm{x}_{T}) , which follows a Gaussian distribution approximately. However, directly sampling from the reverse SDE requires a large number of discretization steps and is typically very slow. To accelerate the sampling process, prior work (e.g., ( Song et al., 2020b ; Lu et al., 2022a ) leveraged the relation between the above SDE and ODE and designed ODE solvers for sampling. In particular, it is known that for SDE (Eq. 20 ), the following ordinary differential equation (ODE), called the Probability Flow ODE (PF-ODE), has the same marginal distribution q t ​ ( 𝒙 ) q_{t}(\bm{x}) ( Song et al., 2020b ; Lu et al., 2022a ) : d ​ 𝒙 t d ​ t = f ⁡ ( t ) ​ 𝒙 t − 1 2 ​ g 2 ​ ( t ) ​ ∇ 𝒙 ​ log ⁡ q t ​ ( 𝒙 t ) , 𝒙 T ∼ q T ​ ( 𝒙 T ) \frac{\mathrm{d}\bm{x}_{t}}{\mathrm{~d}t}=f(t)\bm{x}_{t}-\frac{1}{2}g^{2}(t)\nabla_{\bm{x}}\log q_{t}\left(\bm{x}_{t}\right),\quad\bm{x}_{T}\sim q_{T}\left(\bm{x}_{T}\right) (21) The term − ∇ log ⁡ q t ​ ( 𝒙 t ) -\nabla\log q_{t}(\bm{x}_{t}) in Eq. 21 is typically called the score function of q t ​ ( 𝒙 t ) q_{t}(\bm{x}_{t}) . In diffusion models, we train the noise prediction model ϵ θ ​ ( 𝒙 t , t ) \bm{\epsilon}_{\theta}(\bm{x}_{t},t) to fit the scaled score function, via minimizing the following score matching objective: ℒ ⁡ ( θ ) \displaystyle\mathcal{L}(\theta) = 𝔼 t ∈ [ 0 , T ] , x t ∼ q t [ w ( t ) | | ϵ θ ( 𝒙 t , t ) + σ ( t ) ∇ log q t ( 𝒙 t ) ) | | 2 ] \displaystyle=\mathbb{E}_{t\in[0,T],x_{t}\sim q_{t}}\left[w(t)||\bm{\epsilon}_{\theta}(\bm{x}_{t},t)+\sigma(t)\nabla\log q_{t}(\bm{x}_{t}))||^{2}\right] = 𝔼 t ∈ [ 0 , T ] , 𝒙 0 ∼ q 0 , ϵ ​ [ w ⁡ ( t ) ​ ‖ ϵ θ ​ ( 𝒙 t , t ) − ϵ ‖ 2 ] \displaystyle=\mathbb{E}_{t\in[0,T],\bm{x}_{0}\sim q_{0},\bm{\epsilon}}\left[w(t)||\bm{\epsilon}_{\theta}(\bm{x}_{t},t)-\bm{\epsilon}||^{2}\right] (22) where w ⁡ ( t ) w(t) is the weight function, ϵ ∼ N ⁡ ( 0 , I ) \bm{\epsilon}\sim N(0,I) and 𝒙 t = α ⁡ ( t ) ​ 𝒙 0 + σ ⁡ ( t ) ​ ϵ \bm{x}_{t}=\alpha(t)\bm{x}_{0}+\sigma(t)\bm{\epsilon} . By substituting the score function with the noise prediction model in Eq. 21 , we obtain the following ODE, which can be used for sampling: d ​ 𝒙 t d ​ t = f ⁡ ( t ) ​ 𝒙 t + g 2 ​ ( t ) 2 ​ σ t ​ ϵ θ ​ ( 𝒙 t , t ) , 𝒙 T ∼ 𝒩 ⁡ ( 𝟎 , σ ~ 2 ​ 𝑰 ) . \frac{\mathrm{d}\bm{x}_{t}}{\mathrm{~d}t}=f(t)\bm{x}_{t}+\frac{g^{2}(t)}{2\sigma_{t}}\bm{\epsilon}_{\theta}\left(\bm{x}_{t},t\right),\quad\bm{x}_{T}\sim\mathcal{N}\left(\mathbf{0},\tilde{\sigma}^{2}\bm{I}\right). (23)

### A.2 More Details on Consistency Models in ( Song et al., 2023 )

In this subsection, we provide more details on the consistency models and consistency distillation algorithm in ( Song et al., 2023 ) . The pre-trained diffusion model used in ( Song et al., 2023 ) adopts the continuous noise schedule from EDM ( Karras et al., 2022 ) , therefore the PF-ODE in Eq. 23 can be simplified as: d ​ 𝐱 t d ​ t = − t ∇ log q t ( 𝒙 t ) ≈ − t 𝒔 ϕ ( 𝐱 t , t ) , \frac{\mathrm{d}\mathbf{x}_{t}}{\mathrm{~d}t}=-t\nabla\log q_{t}(\bm{x}_{t})\approx-t\bm{s}_{\phi}\left(\mathbf{x}_{t},t\right), (24) where the 𝒔 ϕ ​ ( 𝐱 t , t ) ≈ ∇ log ⁡ q t ​ ( 𝒙 t ) \bm{s}_{\phi}\left(\mathbf{x}_{t},t\right)\approx\nabla\log q_{t}(\bm{x}_{t}) is a score prediction model trained via score matching ( Hyvärinen & Dayan, 2005 ; Song & Ermon, 2019 ) . Note that different noise schedules result in different PF-ODE and the PF-ODE in Eq. 24 corresponds to the EDM noise schedule ( Karras et al., 2022 ) . We denote the one-step ODE solver applied to PF-ODE in Eq. 24 as Φ ⁡ ( 𝒙 t , t , ϕ ) \Phi({\bm{x}}_{t},t;\phi) . One can either use Euler ( Song et al., 2020b ) or Heun solver ( Karras et al., 2022 ) as the numerical ODE solver. Then, we use the ODE solver to estimate the evolution of a sample 𝒙 t n \bm{x}_{t_{n}} from 𝒙 t n + 1 \bm{x}_{t_{n+1}} as: 𝒙 ^ t n ϕ ← 𝒙 t n + 1 + ( t n − t n + 1 ) ​ Φ ​ ( 𝒙 t n + 1 , t n + 1 , ϕ ) . \hat{{\bm{x}}}_{t_{n}}^{\phi}\leftarrow{\bm{x}}_{t_{n+1}}+(t_{n}-t_{n+1})\Phi({\bm{x}}_{t_{n+1}},t_{n+1};\phi). (25) ( Song et al., 2020b ) used the same time schedule as in ( Karras et al., 2022 ) : t i = ( ϵ 1 / ρ + i − 1 N − 1 ​ ( T 1 / ρ − ϵ 1 / ρ ) ) ρ t_{i}=(\epsilon^{1/\rho}+\frac{i-1}{N-1}(T^{1/\rho}-\epsilon^{1/\rho}))^{\rho} , and ρ = 7 \rho=7 . To enforce the self-consistency property in Eq. 4 , we maintain a target model 𝜽 − {\bm{\theta}}^{-} , which is updated with exponential moving average (EMA) of the parameter 𝜽 {\bm{\theta}} we intend to learn, i.e., 𝜽 − ← μ ​ 𝜽 − + ( 1 − μ ) ​ 𝜽 {\bm{\theta}}^{-}\leftarrow\mu{\bm{\theta}}^{-}+(1-\mu){\bm{\theta}} , and define the consistency loss as follows: ℒ ⁡ ( 𝜽 , 𝜽 − , Φ ) = 𝔼 𝒙 , t ​ [ d ⁡ ( 𝒇 𝜽 ​ ( 𝒙 t n + 1 , t n + 1 ) , 𝒇 𝜽 − ​ ( 𝒙 ^ t n ϕ , t n ) ) ] , \mathcal{L}({\bm{\theta}},{\bm{\theta}}^{-};\Phi)=\mathbb{E}_{\bm{x},t}\left[d\left({\bm{f}}_{\bm{\theta}}({\bm{x}}_{t_{n+1}},t_{n+1}),{\bm{f}}_{{\bm{\theta}}^{-}}(\hat{{\bm{x}}}^{\phi}_{t_{n}},t_{n})\right)\right], (26) where d ⁡ ( ⋅ , ⋅ ) d(\cdot,\cdot) is a chosen metric function for measuring the distance between two samples, e.g., the squared ℓ 2 \ell_{2} distance d ⁡ ( 𝒙 , 𝒚 ) = ‖ 𝒙 − 𝒚 ‖ 2 2 d({\bm{x}},{\bm{y}})=||{\bm{x}}-{\bm{y}}||_{2}^{2} . The pseudo-code for consistency distillation in Song et al. (2023) . is presented in Algorithm 2 . In their original paper, an Euler solver was used as the ODE solver for the continuous-time setting.

## Appendix B Multistep Latent Consistency Sampling

Now, we present the multi-step sampling algorithm for latent consistency model. The sampling algorithm for LCM is very similar to the one in consistency models ( Song et al., 2023 ) except the incorporation of classifier-free guidance in LCM. Unlike multi-step sampling in diffusion models, in which we predict 𝒛 t − 1 {\bm{z}}_{t-1} from 𝒛 t {\bm{z}}_{t} , the latent consistency models directly predicts the origin 𝒛 0 {\bm{z}}_{0} of augmented PF-ODE trajectory (the solution of the augmented of PF-ODE), given guidance scale ω \omega . This generates samples in a single step. The sample quality can be improved by alternating the denoising and noise injection steps. In particular, in the n n -th iteration, we first perform noise-injecting forward process to the previous predicted sample 𝒛 {\bm{z}} as 𝒛 ^ τ n ∼ 𝒩 ⁡ ( α ⁡ ( τ n ) ​ 𝒛 , σ 2 ​ ( τ n ) ​ 𝐈 ) \hat{{\bm{z}}}_{\tau_{n}}\sim\mathcal{N}(\alpha(\tau_{n}){\bm{z}};\sigma^{2}(\tau_{n})\mathbf{I}) , where τ n \tau_{n} is a decreasing sequence of time steps. This corresponds to going back to point 𝒛 ^ τ n \hat{{\bm{z}}}_{\tau_{n}} on the PF-ODE trajectory. Then, we perform the next 𝒛 0 {\bm{z}}_{0} prediction again using the trained latent consistency function. In our experiments, one can see the second iteration can already refine the generation quality significantly, and high quality images can be generated in just 2-4 steps. We provide the pseudo-code in Algorithm 3 .

## Appendix C Algorithm Details of Latent Consistency Fine-tuning