ImageReward: Learning and Evaluating Human Preferences for Text-to-Image Generation — part 1/5

Cleaned-text code points [0, 15911). Bibliography omitted here; source.md includes it.

# ImageReward: Learning and Evaluating Human Preferences for Text-to-Image Generation

###### Abstract

We present a comprehensive solution to learn and improve text-to-image models from human preference feedback. To begin with, we build ImageReward—the first general-purpose text-to-image human preference reward model—to effectively encode human preferences. Its training is based on our systematic annotation pipeline including rating and ranking, which collects 137k expert comparisons to date. In human evaluation, ImageReward outperforms existing scoring models and metrics, making it a promising automatic metric for evaluating text-to-image synthesis. On top of it, we propose Reward Feedback Learning (ReFL), a direct tuning algorithm to optimize diffusion models against a scorer. Both automatic and human evaluation support ReFL’s advantages over compared methods. All code and datasets are provided at https://github.com/THUDM/ImageReward .

### 1 Introduction

Text-to-image generative models, including auto-regressive [ 43 ; 11 ; 14 ; 16 ; 12 ; 63 ] and diffusion-based [ 37 ; 45 ; 42 ; 46 ] approaches, have experienced rapid advancements in recent years. Given appropriate text descriptions (i.e., prompts), these models can generate high-fidelity and semantically-related images on a wide range of topics, attracting significant public interest in their potential applications and impacts.

Despite the progress, existing self-supervised pre-trained [ 33 ] generators are far from perfect. A primary challenge lies in aligning models with human preference , as the pre-training distribution is noisy and differs from the actual user-prompt distributions. The inherent discrepancy leads to several well-documented issues in the generated images [ 15 ; 31 ] , including but not limited to:

• Text-image Alignment : failing to accurately depict all the numbers, attributes, properties, and relationships of objects described in text prompts, as shown in Figure 1 (a)(b).

• Body Problem : presenting distorted, incomplete, duplicated, or abnormal body parts (e.g., limbs) of humans or animals, as illustrated in Figure 1 (e)(f).

• Human Aesthetic : deviating from the average or mainstream human preference for aesthetic styles, as demonstrated in Figure 1 (c)(d).

• Toxicity and Biases : featuring content that is harmful, violent, sexual, discriminative, illegal, or causing psychological discomfort, as depicted in Figure 1 (f).

These prevalent challenges, however, are difficult to address solely through improvements in model architectures and pre-training data.

In natural language processing (NLP), researchers have employed reinforcement learning from human feedback (RLHF) [ 55 ; 36 ; 39 ] to guide large language models [ 6 ; 7 ; 66 ; 48 ; 64 ] towards human preferences and values. The approach relies on learning a reward model (RM) to capture human preference from massive expert-annotated model output comparisons. Effective though it is, the annotation process can be costly and challenging [ 39 ] , as it requires months of effort to establish labeling criteria, recruit and train experts, verify responses, and ultimately produce the RM.

Contributions. Recognizing the importance of addressing these challenges in generative models, we present and release the first general-purpose text-to-image human preference RM—ImageReward—which is trained and evaluated on 137k pairs of expert comparisons in total, based on real-world user prompts and corresponding model outputs. Based on the effort, we further investigate the direct optimization approach ReFL for improving diffusion generative models. Our main contributions are:

• We systematically identify the challenges for text-to-image human preference annotation, and consequently design a pipeline tailored for it, establishing criteria for quantitative assessment and annotator training, optimizing labeling experience, and ensuring quality validation. We build the text-to-image comparison dataset for training the ImageReward model based on the pipeline. The overall architecture is depicted in Figure 2 .

• We demonstrate that ImageReward outperforms existing text-image scoring methods, such as CLIP [ 41 ] (by 38.6%), Aesthetic [ 50 ] (by 39.6%), and BLIP [ 26 ] (by 31.6%), in terms of understanding human preference in text-to-image synthesis through extensive analysis and experiments. ImageReward is also proven to significantly mitigate the aforementioned issues, providing valuable insights into how human preference can be integrated into generative models.

• We suggest that ImageReward could serve as a promising automatic text-to-image evaluation metric. Compared to FID [ 18 ] and CLIP scores on prompts from real users and MS-COCO 2014, ImageReward aligns consistently to human preference ranking and presents higher distinguishability across models and samples.

• We propose Reward Feedback Learning (ReFL) for tuning diffusion models regarding human preference scorers. Our unique insight on ImageReward’s quality identifiability at latter denoising steps allows the direct feedback learning on diffusion models, which offer no likelihood for their generations. Extensive automatic and human evaluations demonstrate ReFL’s advantages over existing approaches including data augmentation [ 61 ; 13 ] and loss reweighing [ 23 ] .

### 2 ImageReward: Learning to Score and Evaluate Human Preferences

ImageReward is constructed using a systematic pipeline involving data collection and human annotation from experts. Based on the pipeline, we implement the RM training and derive the ImageReward.

#### 2.1 Annotation Pipeline Design

Prompt Selection and Image Collection. The dataset utilizes a diverse selection of real user prompts from DiffusionDB [ 58 ] , an open-sourced dataset. To ensure diversity in selected prompts, we employ a graph-based algorithm that leverages language model-based prompt similarity [ 56 ; 44 ; 53 ] . This selection yields 10,000 candidate prompts, each accompanied by 4 to 9 sampled images from DiffusionDB, resulting in 177,304 candidate pairs for labeling (Cf. Appendix A.1 for details).

Human Annotation Design. Our annotation pipeline involves a prompt annotation stage, which includes categorizing prompts and identifying problematic ones, and a text-image rating stage, where images are rated based on alignment , fidelity , and harmlessness . Subsequently, annotators rank the images in order of preference. To manage potential contradictions in the ranking, we provide trade-offs in our annotation document (completely attached in Appendix B ). Our annotation system is composed of three stages: Prompt Annotation, Text-Image Rating, and Image Ranking. Screenshots of our system are provided in Figure 8 . Annotators were recruited in collaboration with a professional data annotation company, with a majority having at least college-level education. They are trained using documents that describe the labeling process and criteria. To ensure quality, we employ quality inspectors to double-check each annotation, with invalid annotations reassigned for relabeling. Due to the page limits, please refer to Appendix A.3 , A.2 , B for comprehensive details and discussion.

Human Annotation Analysis. After 2 months of annotation, we collected valid annotations for 8,878 prompts, resulting in 136,892 compared pairs. A comprehensive analysis of these prompts, annotations, and challenges discovered is discussed in detail in Appendix A.4 .

#### 2.2 RM Training

Admittedly, human evaluation is after all the touchstone for human preference for synthesized images; but it is limited by labor costs and hard to scale up. We aim to model human preference based on annotations, which can lead to a virtual evaluator free from dependence on humans.

Similar to RM training for language model of previous works [ 55 ; 39 ] , we formulate the preference annotations as rankings. We have k ∈ [ 4 , 9 ] k\in[4,9] images ranked for the same prompt T T (the best to the worst are denoted as x 1 , x 2 , … , x k x_{1},x_{2},...,x_{k} ) and get at most C k 2 C_{k}^{2} comparison pairs if no ties between two images. For each comparison, if x i x_{i} is better and x j x_{j} is worse, the loss function can be formulated as: loss ​ ( θ ) = − 𝔼 ( T , x i , x j ) ∼ 𝒟 ​ [ log ( σ ⁡ ( f θ ​ ( T , x i ) − f θ ​ ( T , x j ) ) ) ] \begin{split}\textrm{loss}(\theta)=-\mathbb{E}_{(T,x_{i},x_{j})\sim\mathcal{D}}[\mathop{\log}(\sigma(f_{\theta}(T,x_{i})-f_{\theta}(T,x_{j})))]\end{split} (1) where f θ ​ ( T , x ) f_{\theta}(T,x) is a scalar value of preference model for prompt T T and generated image x x .

Training Techniques. We use BLIP [ 26 ] as the backbone of ImageReward, as it outperforms conventional CLIP (Cf. Table 2(b) ) in our preliminary experiments. We extract image and text features, combine them with cross attention, and use an MLP to generate a scalar for preference comparison.

Training ImageReward is of no ease. We observe rapid convergence and consequent overfitting, which harms its performance. To address this, we freeze some backbone transformer layers’ parameters, finding that a proper number of fixed layers improves ImageReward’s performance (Cf. Section 4.1 ). ImageReward also exhibits sensitivity to training hyperparameters, such as learning rate and batch size. We perform a careful grid search based on the validation set to determine optimal values.

#### 2.3 As Metric: Re-Evaluating Human Preferences on Text-to-Image Models

Training text-to-image generative models is hard, but evaluating these models reasonably is even harder. In literature [ 11 ; 42 ; 12 ; 46 ] , it has been a de facto practice to evaluate text-to-image generative models on MS-COCO [ 28 ] image-caption dataset against the real images, using fine-tuned or zero-shot FID [ 18 ] scores following DALL-E [ 43 ] setting. Nevertheless, it remains quite dubious whether the FID really fits the current need [ 38 ] , especially from the following aspects:

1. Zero-shot Usage : As generative models are now dominantly used by the public in a zero-shot manner without fine-tuning, fine-tuned FID may not honestly reflect models’ actual performance in real use. In addition, despite the adoption of zero-shot FID in recent trends, the possible leak of MS-COCO in some models’ pre-training data would make it a potentially unfair setting.

2. Human Preference : FID measures the average distance between generated images and reference real images, and thus fails to encompass human preference that is crucial to text-to-image synthesis in evaluation. Moreover, FID’s relies on average over the whole dataset to provide an accurate assessment, whereas in many cases we need the metric to serve as a selector over single images.

Seeing these challenges, we propose ImageReward as a promising zero-shot automatic evaluation metric for text-to-image model comparison and individual sample selection.

Better Human Alignment Across Models. We conduct researcher annotation (i.e., by authors) across 6 popular high-resolution (around 512 × \times 512) available text-to-image models: CogView 2 [ 12 ] , Versatile Diffusion (VD) [ 62 ] , Stable Diffusion (SD) 1.4 and 2.1-base [ 45 ] , DALL-E 2 (via OpenAI API) [ 42 ] , and Openjourney 1 1 1 https://openjourney.art/ , to identify the alignment of different metrics to human.

We sample 100 real-user test prompts for the alignment test, with each model generating 10 outputs as candidates. To compare these models, we first pick the best image out of 10 outputs by each model on each prompt. Then, the annotators rank the images from different models for each prompt, following the disciplines for ranking described in Section 2.1 . We aggregate all annotators’ annotations, and compute the final win count of each model to all others (Cf. Table 1 ).

For ImageReward and CLIP scores, we report their average for 1,000 text-image pairs per model. We also document all models’ zero-shot FID and ImageReward score (30k) on MS-COCO 2014 valid set following prior practices [ 42 ; 12 ] , where outputs are unified to 256 × \times 256 resolution and optimal classifier-free guidance values are selected by grid search (i.e., [1.5, 2.0, 3.0, 4.0, 5.0]). As shown in Table 1 , ImageReward aligns well with human ranking, whereas zero-shot FID and CLIP are not.

Better Distinguishability Across Models and Samples. Another highlight is that, compared to CLIP, we observe that ImageReward can better distinguish the quality between individual samples. Figure 3 presents a box plot of ImageReward and CLIP’s score distributions on the 1,000 generations per model. The distributions are normalized to 0.0 to 1.0 using minimum and maximum values of ImageReward and CLIP scores per model, and outliers are discarded. As it demonstrates, ImageReward’s scores in each model have a much larger interquartile range than that of CLIP, which means ImageReward can well distinguish the quality of images from each other. Besides, in terms of comparison across models, we discover that the medians of the ImageReward scores are also roughly in line with human ranking in Table 1 . On the contrary, CLIP’s medians fail to present the property.

### 3 ReFL: Reward Feedback Learning Improves Text-to-Image Diffusion

Though ImageReward can pick out highly human-preferred images from many generations of a prompt, the generate-and-then-filter paradigm could be expensive and inefficient in practical applications. Therefore, we seek to improve text-to-image generative models, particularly for the popular latent diffusion models, for allowing high-quality generation in single or very few trials.

Challenge. In NLP, researchers have reported using reinforcement learning algorithms (e.g., PPO [ 51 ] ) to steer language models to align to human preference [ 55 ; 36 ; 39 ] , which depends on the likelihood of a whole generation to update the model.

However, unlike language models, latent diffusion models (LDMs)’s multi-step denoising generation cannot yield likelihoods for their generations, and thus fail to adopt the same RLHF approaches. A potentially similar approach is classifier-guidance [ 54 ; 9 ] technique during LDM inference. Nonetheless, it is for inference only and employs a classifier necessarily trained on noisy intermediate latents, which naturally contradicts RMs’ annotation where images need to be completely denoised for humans to mark correct preference. Some concurrent works propose some alternative indirect solutions, such as using RMs to filter dataset for fine-tuning [ 61 ; 13 ] , or to re-weight losses of training samples according to their qualities [ 23 ] . Nevertheless, these data-oriented approaches are virtually indirect. They could rely heavily on proper fine-tuning data distributions and finally only improve the LDMs mildly.

ReFL: Insight and Solution. We endeavor to develop a direct optimization method for improving LDMs according to an RM (e.g., ImageReward). Looking into ImageReward scores along denoising steps (i.e., 40 in our case), we derive an intriguing insight (Cf. Figure 4 ) that when we directly predict x t → x 0 ′ x_{t}\to x_{0}^{\prime} at a step t t (different from the real latent x 0 x_{0} which experiences x t → x t + 1 → … → x 0 x_{t}\to x_{t+1}\to...\to x_{0} ):

• When t ≤ 15 t\leq 15 : ImageReward scores for all generations are unanimously low.

• When 15 ≤ t ≤ 30 15\leq t\leq 30 : High-quality generations begin to stand out, but overall we still cannot clearly judge all generations’ final qualities based on the current ImageReward scores.

• When t ≥ 30 t\geq 30 : Generations of different ImageReward scores are generally distinguishable.

In light of the observation, we conclude that ImageReward scores for generations x 0 ′ x_{0}^{\prime} after 30 steps of denoising, unnecessarily the final step, could serve as reliable feedback for improving LDMs.