Emu: Enhancing Image Generation Models Using Photogenic Needles in a Haystack — part 2/2

Cleaned-text code points [15636, 27168). Bibliography omitted here; source.md includes it.

4. Subject and Background. The image should have a sense of depth between the foreground and background elements. The background should be uncluttered but not overly simplistic or dull. The focused subjects must be intentionally placed within the frame, ensuring that all critical details are clearly visible without compromise. For instance, in a portrait, the primary subject of image should not extend beyond the frame or be obstructed. Furthermore, the level of detail on the foreground subject is extremely important.

5. Additional Subjective Assessments. Furthermore, we request annotators to provide their subjective assessments to ensure that only images of exceptionally aesthetic quality are retained by answering a couple of questions, such as: (i) Does this image convey a compelling story? (ii) Could it have been captured significantly better? (iii) Is this among the best photos you’ve ever seen for this particular content?

Through this filtering process, we retained a total of 2000 exceptionally high-quality images. Subsequently, we composed ground-truth captions for each of them. Note that some of these handpicked images are below our target resolution of 1024 × 1024 1024\times 1024 . We trained a pixel diffusion upsampler inspired by the architecture proposed in [ 30 ] to upsample these images when necessary.

### 3.4 Quality-Tuning

We can think of the visually stunning images (like the 2000 images we collected) as a subset of all images that share some common statistics. Our hypothesis is that a strongly pre-trained model is already capable of generating highly aesthetic images, but the generation process is not properly guided towards always producing images with these statistics. Quality-tuning effectively restricts outputs to a high-quality subset.

We fine-tune the pre-trained model with a small batch size of 64 64 . We use a noise-offset of 0.1 0.1 at this stage. Note that early stopping is important here as fine-tuning on a small dataset for too long will result in significant overfitting and degradation in generality of visual concepts. We fine-tune for no more than 15 15 K iterations despite the loss still decreasing. This total iteration number is determined empirically.

## 4 Experiments

We compare our quality-tuned model to our pre-trained model to demonstrate the effectiveness of quality-tuning. To place the visual appeal of our generated images in context with a current state-of-the-art model, we compare our model to SDXLv1.0 [ 21 ] . Due to lack of access to training data of SDXL and their underlying model, we leveraged their corresponding APIs for our comparison. Note that unlike SDXL, we use a single stage architecture, and do not use a subsequent refinement stage. As stated earlier, we also show that quality-tuning is not specific to LDMs, and can be applied to other architectures – pixel diffusion and masked generative transformer models.

### 4.1 Evaluation Setting

#### Prompts.

We evaluate on two large sets of prompts: 1600 PartiPrompts [ 37 ] which is commonly used for text-to-image generation benchmarking, and our 2100 Open User Input (OUI) Prompts. The OUI prompt set is based on real-world user prompts. It captures prompts that are popular with text-to-image models, reflecting visual concepts that are relevant to real-world use cases (Figure 5 ), paraphrased by LLMs to be closer to how regular users might input prompts (as opposed to being highly prompt engineered). The overall motivation was to capture the creativity and intricacies of popular prompts for text-to-image models so we are pushing the capabilities of models, while also being grounded in likely real world use cases.

#### Metrics.

We use two separate evaluation metrics: visual appeal and text faithfulness. Visual appeal refers to the overall aesthetic quality of a generated image. It combines various visual elements such as color, shape, texture, and composition that creates a pleasing and engaging look. The concept of visual appeal is subjective and varies from person to person, as what may be aesthetically pleasing to one person may not be to another. Therefore, we ask five annotators to rate each sample. Concretely, we show annotators two images A and B, side-by-side, each generated by a different model using the same caption. The text captions are not displayed. Annotators choose which image is more visually appealing by selecting “A”, “B” or “Tie”.

Text faithfulness refers to the degree of similarity between a generated image and a text caption. In this task, we again display two generated images A and B, side-by-side, but with the caption alongside the images. The annotators are asked to ignore the visual appeal of the images and choose which ones best describe the caption with choices “A ”, “B”, “Both”, and “Neither”, where “Both” and “Neither” are considered as “Tie”. In this task, we have three annotators to annotate each sample pair.

We do not report “standard” metrics such as FID scores. As argued in many recent papers ( e.g., [ 21 , 18 ] ), FID scores do not correlate well with human assessment of the performance of generative models.

### 4.2 Results

#### Effectiveness of Quality-Tuning.

First, we compare our quality-tuned model, Emu, with the pre-trained model. See Figure 7 for random (not cherry-picked) qualitative examples before and after quality-tuning. Note the highly aesthetic non-photorealistic image as well, validating our hypothesis that following certain photography principles in curating the quality-tuning dataset leads to improved aesthetics for a broad set of styles. We show more examples of generated images using Emu in Figure 8 and Figure 9 .

Quantitatively, as shown in Figure 6 (top), after quality-tuning, Emu is preferred in both visual appeal and text faithfulness by a significant margin. Specifically, Emu is preferred 82.9 % 82.9\% and 91.2 % 91.2\% of the time for visual appeal, and 36.7 % 36.7\% and 47.9 % 47.9\% of the time for text faithfulness on PartiPrompts and OUI Prompts, respectively. In contrast, the pre-trained model is preferred only 15.4 % 15.4\% and 7.9 % 7.9\% of the time for visual appeal, and 21.0 % 21.0\% and 18.5 % 18.5\% of the time for text faithfulness on PartiPrompts and OUI Prompts. The remaining cases result in ties. From the two large sets of evaluation data that covers various domains and categories, we did not observe degradation of generality across visual concepts. In fact, as seen, text-faithfulness also improved. We hypothesize this is because the captions of the 2000 quality-tuning images were manually written, while the captions in the pre-training dataset tend to be noisy. Finally, we analyze the results on non-photorealistic stylized prompts ( e.g., sketches, cartoons, etc.). We find that the improvements broadly apply to a variety of styles, see Figure 6 (bottom).

#### Visual Appeal in the Context of SoTA.

To place the visual appeal of our generated images in the context of current state-of-the-art, we compare Emu with SDXLv1.0 [ 21 ] . As shown in Table 2 , our model is preferred over SDXLv1.0 in visual appeal by a significant margin – including on stylized (non-photorealistic) prompts.

#### Quality-Tuning Other Architectures.

Next, we show our quality-tuning can also improve other popular architectures, such as pixel diffusion and masked generative transformer models. Specifically, we re-implement and re-train from scratch a pixel diffusion model, similar to Imagen [ 30 ] architecture, and a masked generative transformer model, similar to Muse [ 9 ] architecture, and then quality-tune them on 2000 images. We evaluate both quality-tuned models on 1 / 3 1/3 randomly sampled PartiPrompts. As shown in Figure 10 , both architectures show significant improvement after quality-tuning on both visual appeal and text faithfulness metrics.

### 4.3 Ablation Studies

We do ablation studies on the fine-tuning dataset with a focus on visual appeal. We first investigate the impact of the dataset size. We report results of quality-tuning on randomly sampled subsets of sizes 100, 1000 and 2000 in Table 3 . With even just 100 fine-tuning images, the model can already be guided towards generating visually-pleasing images, jumping from a win rate of 24.8 % 24.8\% to 60 % 60\% compared with SDXL.

## 5 Limitation

Limitation of Human Evaluation. Relative to most published works, the prompt sets we evaluate on are reasonably large (1600 Parti prompts and our 2100 Open User Input prompts), under a multi-review annotation setting. Even then, the evaluation results may not fully reflect real-world usage of the models. In addition, human evaluation of text-to-image models, especially when it comes to aesthetics, is inherently subjective and noisy. As a result, evaluation on a different set of prompts or with different annotators and guidelines may lead to different results.

Limitations of Small-Scale Fine-Tuning. The role of quality-tuning is to restrict the output distribution to a high-quality domain. However, issues rooted from pre-training may still persist. For instance, the models may struggle to generate certain objects that were not sufficiently learned during pre-training.

Limitations of Text-to-Image Models in General. Like other text-to-image models, our models may generate biased, misleading, or offensive outputs. We’ve invested a significant amount in fairness and safety of our models - starting with balanced dataset construction, creating dedicated evaluation sets for these risk categories and investing multiple hours of redteaming.

## 6 Conclusion

In this paper, we demonstrated that manually selecting high quality images that are highly aesthetically-pleasing is one of the most important keys to improving the aesthetics of images generated by text-to-image generative models. We showed that with just a few hundred to thousand fine-tuning images, we were able to improve the visual appeal of generated images without compromising on the generality of visual concepts depicted. With this finding, we build Emu, a LDM for high-quality image synthesis. On the commonly used PartiPrompts and our Open User Input Prompts, we carefully evaluate our model against a publicly available state-of-the-art text-to-image model (SDXLv1.0 [ 21 ] ) as well as our pre-trained LDM. We also show that quality-tuning not only improves LDMs, but also pixel diffusion and masked generative transformer models.

## 7 Acknowledgement

This work would not have been possible without a large group of collaborators who helped with the underlying infrastructure, data, privacy, and the evaluation framework. We extend our gratitude to the following people for their contributions (alphabetical order): Eric Alamillo, Andrés Alvarado, Giri Anantharaman, Stuart Anderson, Snesha Arumugam, Chris Bray, Matt Butler, Anthony Chen, Lawrence Chen, Jessica Cheng, Lauren Cohen, Jort Gemmeke, Freddy Gottesman, Nader Hamekasi, Zecheng He, Jiabo Hu, Praveen Krishnan, Carolyn Krol, Tianhe Li, Mo Metanat, Vivek Pai, Guan Pang, Albert Pumarola, Ankit Ramchandani, Stephen Roylance, Kalyan Saladi, Artsiom Sanakoyeu, Dev Satpathy, Alex Schneidman, Edgar Schoenfeld, Shubho Sengupta, Hardik Shah, Shivani Shah, Yaser Sheikh, Karthik Sivakumar, Lauren Spencer, Fei Sun, Ali Thabet, Mor Tzur, Mike Wang, Mack Ward, Bichen Wu, Seiji Yamamoto, Licheng Yu, Hector Yuen, Luxin Zhang, Yinan Zhao, and Jessica Zhong.

Finally, thank you Connor Hayes, Manohar Paluri and Ahmad Al-Dahle for your support and leadership.