Emu: Enhancing Image Generation Models Using Photogenic Needles in a Haystack — part 1/2

Cleaned-text code points [0, 15636). Bibliography omitted here; source.md includes it.

# Emu: Enhancing Image Generation Models Using Photogenic Needles in a Haystack

###### Abstract

Training text-to-image models with web scale image-text pairs enables the generation of a wide range of visual concepts from text. However, these pre-trained models often face challenges when it comes to generating highly aesthetic images. This creates the need for aesthetic alignment post pre-training. In this paper, we propose quality-tuning to effectively guide a pre-trained model to exclusively generate highly visually appealing images, while maintaining generality across visual concepts. Our key insight is that supervised fine-tuning with a set of surprisingly small but extremely visually appealing images can significantly improve the generation quality. We pre-train a latent diffusion model on 1.1 1.1 billion image-text pairs and fine-tune it with only a few thousand carefully selected high-quality images. The resulting model, Emu, achieves a win rate of 82.9 % 82.9\% compared with its pre-trained only counterpart. Compared to the state-of-the-art SDXLv1.0, Emu is preferred 68.4 % 68.4\% and 71.3 % 71.3\% of the time on visual appeal on the standard PartiPrompts and our Open User Input benchmark based on the real-world usage of text-to-image models. In addition, we show that quality-tuning is a generic approach that is also effective for other architectures, including pixel diffusion and masked generative transformer models.

## 1 Introduction

Recent advances in generative models have enabled them to generate various high-quality content, such as text [ 2 , 33 ] , image [ 25 , 21 ] , music [ 16 ] , video [ 32 ] , and even 3D scenes [ 22 , 19 , 34 ] , which has fundamentally revolutionized generative artificial intelligence (AI). In this paper, we present a recipe we have found to be effective for training highly aesthetic text-to-image models. It involves two stages: a knowledge learning stage, where the goal is to acquire the ability to generate virtually anything from text, which typically involves pre-training on hundreds of millions of image-text pairs; and a quality learning stage, which is necessary to restrict the output to a high-quality and aesthetically pleasing domain. We refer to the process of fine-tuning for the purpose of improving quality and promoting aesthetic alignment as quality-tuning for short.

Our key insight is that to effectively perform quality-tuning, a surprisingly small amount – a couple of thousand – exceptionally high-quality images and associated text is enough to make a significant impact on the aesthetics of the generated images without compromising the generality of the model in terms of visual concepts that can be generated. Although having more data while maintaining the same level of quality may be helpful, any attempts to prioritize quantity over quality may result in a compromise of the quality of generated images.

This is an interesting finding in the broader landscape of fine-tuning generative models. Quality-tuning for visual generative models can be thought of as analogous to instruction-tuning for large language models (LLMs) in terms of improving generation quality. First, before instruction-tuning, language models are more prone to generating low-quality text, which may be inconsistent in tone, overly verbose or concise, or simply unhelpful or even toxic [ 20 , 6 , 7 , 35 , 13 ] ; ChatGPT-level [ 2 ] performance is achieved with effective instruction-tuning [ 20 ] . Similarly, we find that quality-tuning significantly improves the generation quality of text-to-image models. Second, the recipe for effectively performing instruction-tuning and quality-tuning is similar: use high-quality data, even if the quantity has to be small to maintain quality. Llama2 [ 33 ] has been fine-tuned on 27 27 K high-quality prompts, which can be considered a very small quantity compared to the billions or trillions of pre-training tokens. Similarly, we find that strong text-to-image performance can be achieved by fine-tuning with even less data – a few thousand carefully selected images. Lastly, the knowledge obtained from pre-training is mostly retained after both instruction-tuning and quality-tuning. Like instruction-tuned LLMs, quality-tuned text-to-image models retain their generality in terms of the visual concepts that can be generated. These post pre-training stages align the knowledge to downstream user value – improving text quality and following instructions in the case of LLMs, and promoting aesthetic alignment in the case of text-to-image models.

Concretely, we pre-train a latent diffusion model (LDM) on 1.1 1.1 billion image-text pairs and quality-tune the model on a few thousand hand-picked exceptionally high-quality images selected from a large corpus of images. By its nature, the selection criterion is subjective and culturally dependent. We follow some common principles in photography, including but not limited to composition, lighting, color, effective resolution, focus, and storytelling to guide the selection process. With a few optimizations to the latent diffusion architecture, we start with a strong pre-trained model and dramatically improve the visual appeal of our generated images through quality-tuning. In fact, it significantly outperform a state-of-the-art publicly available model SDXLv1.0 [ 21 ] on visual appeal. We call our quality-tuned LDM Emu. We show example generations from Emu in Figure 1 and Figure 2 .

Furthermore, we show that quality-tuning is a generic approach that is also effective for pixel diffusion and masked generative transformer models.

Our main contributions are:

• We build Emu, a quality-tuned latent diffusion model that significantly outperforms a publicly available state-of-the-art model SDXLv1.0 on visual appeal.

• To the best of our knowledge, this is the first work to emphasize the importance of a good fine-tuning recipe for aesthetic alignment of text-to-image models. We provide insights and recommendations for a key ingredient of this recipe – supervised fine-tuning with a surprisingly small amount of exceptionally high-quality data can have a significant impact on the quality of the generated images. Image quality should always be prioritized over quantity.

• We show that quality-tuning is a generic approach that also works well for other popular model architectures besides LDM, including pixel diffusion and masked generative transformer models.

## 2 Related Work

#### Text-to-Image Models.

Generating an image from a textual description has been explored using various approaches. Diffusion-based methods learn a denoising process to gradually generate images from pure Gaussian noise [ 14 ] . The denoising process can occur either in pixel space or latent space, resulting in pixel diffusion models [ 25 , 30 , 5 ] or latent diffusion models [ 27 , 21 ] , which feature higher efficiency by reducing the size of spatial features. Generative transformer methods usually train autoregressive [ 26 , 37 , 11 , 10 , 4 , 38 ] or non-autoregressive (masked) [ 9 ] transformers in discrete token space to model the generation process. Generative adversarial network [ 31 , 17 ] also show remarkable capability of generating realistic images. While these models exhibit unprecedented image generation ability, they do not always generate highly aesthetic images.

#### Fine-Tuning Text-to-Image Models.

Given a pre-trained text-to-image model, different methods have been developed to enable specific tasks. A number of techniques have been developed to personalize or adapt text-to-image models to a new subject or style [ 28 , 12 , 15 ] . ControlNet [ 39 ] provides additional control to the generation process by additionally conditioning on pose, sketch, edges, depth, etc. InstructPix2Pix [ 8 ] makes text-to-image models follow editing instructions by fine-tuning them on a set of generated image editing examples. To the best of our knowledge, this is the first work highlighting fine-tuning for generically promoting aesthetic alignment for a wide range of visual domains.

#### Fine-Tuning Language Models.

Fine-tuning has become a critical step in building high-quality LLMs [ 33 , 20 , 1 ] . It generically improves output quality while enabling instruction-following capabilities. Effective fine-tuning of LLMs can be achieved with a relatively small but high-quality fine-tuning dataset, e.g., using 27 27 K prompts in [ 33 ] . In this work, we show that effective fine-tuning of text-to-image models can be also achieved with a small but high-quality fine-tuning dataset. This finding shows an interesting connection between fine-tuning vision and language models in generative AI.

## 3 Approach

As discussed earlier, our approach involves a knowledge learning stage followed by a quality-tuning stage. This may seem like a well-known recipe when mapped to pre-training and fine-tuning. That said, the key insights here are: (i) the fine-tuning dataset can be surprisingly small, on the order of a couple of thousand images, (ii) the quality of the dataset needs to be very high, making it difficult to fully automate data curation, requiring manual annotation, and (iii) even with a small fine-tuning dataset, quality-tuning not only significantly improves the aesthetics of the generated images, but does so without sacrificing generality as measured by faithfulness to the input prompt. Note that the stronger the base pre-trained model, the higher the quality of the generated images after quality-tuning. To this end, we made several modifications to the latent diffusion architecture [ 27 ] to facilitate high-quality generation. That said, quality-tuning is general enough and can be applied to a variety of architectures.

In this section, we first introduce the latent diffusion architecture we use. Then, we discuss the pre-training stage, followed by the protocol for collecting the high-quality fine-tuning dataset, and finally the quality-tuning stage. Later in Section 4 , we demonstrate that quality-tuning is not limited to latent diffusion models but also improve other models such as pixel diffusion [ 30 ] and masked generative transformer [ 9 ] models.

### 3.1 Latent Diffusion Architecture

We design a latent diffusion model that outputs 1024 × 1024 1024\times 1024 resolution images. Following standard latent diffusion architecture design, our model has an autoencoder (AE) to encode an image to latent embeddings and a U-Net to learn the denoising process.

We find that the commonly used 4-channel autoencoder (AE-4) architecture often results in a loss of details in the reconstructed images due to its high compression rate. The issue is especially noticeable in small objects. Intuitively, it compresses the image resolution by 64 × 64\times with three 2 × 2 2\times 2 downsampling blocks but increases the channel size only from 3 (RGB) to 4 (latent channels). We find that increasing the channel size to 16 significantly improves reconstruction quality (see Table 1 ). To further improve the reconstruction performance, we use an adversarial loss and apply a non-learnable pre-processing step to RGB images using a Fourier Feature Transform to lift the input channel dimension from 3 (RGB) to a higher dimension to better capture fine structures. See Figure 3 for qualitative results of autoencoders of different channel size.

We use a large U-Net with 2.8B trainable parameters. We increase the channel size and number of stacked residual blocks in each stage for larger model capacity. We use text embeddings from both CLIP ViT-L [ 23 ] and T5-XXL [ 24 ] as the text conditions.

### 3.2 Pre-training

We curate a large internal pre-training dataset consisting of 1.1 1.1 billion images to train our model. The model is trained with progressively increasing resolutions, similar to [ 21 ] . This progressive approach allows the model to efficiently learn high-level semantics at lower resolutions and improve finer details at the highest resolutions We also use a noise-offset [ 3 ] of 0.02 in the final stage of pre-training. This facilitates high-contrast generation, which contributes to the aesthetics of the generated images.

### 3.3 High-Quality Alignment Data

As discussed before, in order to align the model towards highly aesthetic generations – quality matters significantly more than quantity in the fine-tuning dataset (see Section 4.3 for an ablation study on quality vs quantity). As also discussed, the notion of aesthetics is highly subjective. Here we discuss in detail what aesthetics we chose and how we curated our fine-tuning dataset by combining both automated filtering and manual filtering. The general quality-tuning strategy will likely apply to other aesthetics as well.

Automatic Filtering. Starting from an initial pool of billions of images, we first use a series of automatic filters to reduce the pool to a few hundreds of millions. These filters include but are not limited to offensive content removal, aesthetic score filter, optical character recognition (OCR) word count filter to eliminate images with too much overlaying text on them, and CLIP score filter to eliminate samples with poor image-text alignment, which are standard pre-filtering steps for sourcing large datasets. We then perform additional automated filtering via image size and aspect ratio. Lastly, to balance images from various domains and categories, we leverage visual concept classification [ 36 ] to source images from specific domains ( e.g., portrait, food, animal, landscape, car, etc). Finally, with additional quality filtering based on proprietary signals ( e.g. , number of likes), we can further reduce the data to 200 200 K.

Human Filtering. Next, we perform a two-stage human filtering process to only retain highly aesthetic images. In the first stage, we train generalist annotators to downselect the image pool to 20K images. Our primary goal during this stage is to optimize recall, ensuring the exclusion of medium and low quality that may have passed through the automatic filtering. In the second stage, we engage specialist annotators who have a good understanding of a set of photography principles. Their task is to filter and select images of the highest aesthetic quality (see Figure 4 for examples). During this stage, we focus on optimizing precision, meaning we aim to select only the very best images. A brief annotation guideline for photorealistic images is as follows. Our hypothesis is that following basic principles of high quality photography leads to generically more aesthetic images across a variety of styles, which is validated via human evaluation. 1. Composition. The image should adhere to certain principles of professional photography composition, including the “Rule Of Thirds”, “Depth and Layering”, and more. Negative examples may include imbalance in visual weight, such as when all focal subjects are concentrated on one side of the frame, subjects captured from less flattering angles, or instances where the primary subject is obscured, or surrounding unimportant objects are distracting from the subject.

2. Lighting. We are looking for dynamic lighting with balanced exposure that enhances the image, for example, lighting that originates from an angle, casting highlights on select areas of the background and subject(s). We try to avoid artificial or lackluster lighting, as well as excessively dim or overexposed light.

3. Color and Contrast. We prefer images with vibrant colors and strong color contrast. We avoid monochromatic images or those where a single color dominates the entire frame.