ImageReward: Learning and Evaluating Human Preferences for Text-to-Image Generation — part 4/5

Cleaned-text code points [47535, 63400). Bibliography omitted here; source.md includes it.

Label (drop-down menu) - What is the main object of this text description? Choose the label that seems most appropriate.

The user’s intent is unclear(yes/no) - If it is really unclear what the user is asking the AI system to do, then mark "yes" here. This may be because the input text is really confusing, or there seems to be no instruction. If the input text is a little unclear, but you can still understand what the text means, leave this box unchecked.

- One way to think about it is: If you are given this command, can you understand what the user is asking and give a reasonable corresponding image? If the answer is "no", then mark the box "yes".

(Multiple choice) Does the input text have any of the following issues? - Contains sexual content

- Contains violent content

- Contains content that defames certain groups: By " defame " we mean unfair criticism. Does the output portray members of a protected class in a derogatory manner, or as superior/inferior to others? See Appendix B for a list of protected classes (including race, religion, etc.).

- Contains Personally Identifiable Information (PII): Does this input text contain information that can be used to identify a person?

- There are no such problems

#### B.2 Annotate generated images

For each AI model output, please give the annotation of the following questions.

Overall Rating (Likert scale 1-7) - How good is the generated image of this text (i.e., has an image-text alignment, high fidelity, and harmlessness.)? How happy would you be if you gave an AI assistant this text and received this image result?

- 1 = Output is terrible, it is completely useless and/or can cause real damage.

- 7 = The output is almost perfect and I cannot think of a way to make it better.

Image-text alignment (Likert scale 1-7) - How well do the generated images match the given text?

- 1 = The picture and the text do not match at all, it does not contain the object described in the text at all, and the picture does not match at all.

- 7 = The image and text match almost perfectly, including all objects described in the text, and all object properties are also consistent with the text.

Fidelity (Likert scale 1-7) - The output image should be true to the shape and characteristics that the object should have and will not be generated haphazardly. Some counter-examples of low-fidelity images. • “Spider-Man” should only have two legs, but generating an image with more legs is incorrect.

• “Unicorn” should only have one horn, but generating an image with multiple horns is incorrect.

• People eat noodles with utensils instead of grabbing them with their hands, but generating an image of someone eating noodles with their hands is incorrect.

- 1 = There is an obvious error in the above image, and seriously affects the degree of beauty or coordination of the picture

- 4 = There are obvious mistakes but no harm, the tires in the picture below are obviously separated from the car body is not right, but the overall picture is coordinated.

- 7 = No image errors, beautiful and coordinated pictures

(Multiple choice) Does the image have any of the following issues? - Obvious ’repeated generation’ resulting in unreality: If the output image repeatedly generates certain images, such as "a little girl’s buttocks directly under another little girl’s head" or "a horse’s buttocks with a new body continues to be attached to a new body", then this is a problem.

- Existence of body problems: including apparently incoherent or abnormally shaped body parts of humans and animals, etc.

- Too blurred to see objects

- Causes psychological discomfort: If you feel discomfort when you see this picture, you can mark this question and answer it according to how you feel.

- Output contains sexual content

- The output contains violent content

- Output contains content that defames certain groups: By "defame", we mean unfair criticism. Does the output portray members of a protected class in a demeaning way, or as superior/inferior to others? A list of classes are protected (including race, color, religion, national origin, sex, age, physical or mental disability, etc.).

- There are no such problems

#### B.3 Rank generated images

Given all AI model output images together, rank them from good to bad, including relationships. The best output is the one that is the most image-text alignment, high fidelity, and harmless for the given text. If the images are close in quality, do not hesitate to mark them as tied. Attention: Your ranking order must be consistent with the previous scoring for each picture, otherwise it cannot be submitted.

#### B.4 Frequently asked questions

Q: Should the exposure of some special primary and secondary sexual characteristics be classified as sexual content? Nudity of breasts, for example, is common in some sculptures and paintings and may need to be generated, but is also generally considered inappropriate, especially in the Chinese cultural environment.

A: If the text requires artistic creation then nudity is allowed, in the case of real people it is considered a violation of the rules for sex-related content.

Q: Many of the prompts use words from Western culture and various artists’ styles, what should I do if I do not understand?

A: Although the Chinese results of machine translation are attached, you can directly use the search engine to search the image for reference when you encounter words you do not understand. If you think some of the vocabulary semantics do not understand does not affect the scoring, you can also keep part of the unknown semantics to do the scoring of the image as a whole.

Q: The input contains a command, but it is confusing/obscure. What should I do?

A: You may encounter the following tasks. • The task seems confusing

• You do not feel you know exactly what it means to do this task well

• There are two possible plausible explanations for this task

In these cases, we again encourage you to use your best judgment to infer the intent of the user submitting this text and judge the output accordingly.

Q: When should I skip a mission?

A: There is an option to skip a task if • You are uncomfortable with the task, e.g. it involves gore, horror, pornography, etc.

• You do not think you can do the task well, e.g. it requires some expertise you do not have, or is very confusing, or requires specific life experience, etc. (Note: there is a limit to the number of skips)

### Appendix C More Analysis on ImageReward’s Performance and Properties

#### C.1 Average Scores of the Highest/Lowest Ranked Images

When evaluating several images, we are also concerned about which one is the best or worst, and whether the preference model can pick. Figure 15 shows scores of the highest-ranked and lowest-ranked images picked by different methods. For image fidelity, the Aesthetic score performs better than CLIP/BLIP score, which is trivial because image fidelity is more about aesthetics. ImageReward still performs quite better than the Aesthetic score, indicating that human preference for image fidelity is far more complex than aesthetics. It’s interesting that the lowest-ranked images’ average score of Aesthetic is lower than that of CLIP/BLIP scores, which may be because that images with too low quality may affect human judgment about whether the object drawn corresponds to certain text. Overall, our ImageReward model performs the best. Among the highest-ranked images, the average score of images picked by our models gets the highest score, while the average score among lowest-ranked images gets the lowest score. Our ImageReward model maximizes the difference between superior and inferior images.

#### C.2 Recall/Filter the Best/Worst Image

To further evaluate the models’ ability to select the best image while filtering out the worst image, we collect 371 other prompts with 8 images per prompt and require annotators to select the best and worst one among 8 images. Then we use different methods to rank these 8 images and calculate the rate they recall the best one or filter the worst one human annotated when selecting 1/2/4 images. These statistics are also shown in Table 3 . Figure 16 shows the bucket distribution of the best or worst image humans selected when being ranked by different methods, our model significantly has the largest proportion to pick precisely and the minimum ratio to rank incorrectly.

Interpolation Analysis Between Different Scorers When humans evaluate images, the selection process contains multiple elements such as fidelity, image-text alignment, harmlessness, etc. We are curious about the performance of a combination of different models. We test interpolation among CLIP score, Aesthetic score, and our ImageReward model, their accuracy on the test set can be seen in Figure 17 .

### Appendix D Comparison between ImageReward and Other Reward Models

Besides ImageReward, other reward models aimed at alignment with human preferences have also emerged recently, such as HPS [ 61 ] and PickScore [ 21 ] . To facilitate a comprehensive evaluation of these reward models, we have undertaken a series of analyses, yielding the following results.

#### D.1 Human Evaluation

Human evaluation requires annotators to rank all the images generated from the same prompt in different datasets, and the comparisons were analyzed in Table 5 and 6 , where "Bo64" means "the Best of 64 (Images)", i.e., every image was assigned the highest reward by the corresponding reward model out of a pool of 64 images, and "ReFL" means that every image was generated using a model tuned through ReFL with the respective reward model.

#### D.2 Train Set Distribution

We employed t-SNE [ 57 ] to visualize the prompt distribution of training sets of PickScore and ImageReward. Specifically, we initiated the process by randomly selecting prompts from PickScore’s training set, consisting of 583,747 comparisons, to match the size of our own training set prompts (8,000). Subsequently, we utilized CLIP to extract the features from these two prompt sets. Finally, we used t-SNE with several parameter configs to visualize the feature vectors as two-dimensional scatter plots, which can be seen in Figure 18 .

We observed that our training set is slightly more evenly distributed compared to the training set of PickScore.

### Appendix E Implementation Details of Related LDM Optimization Methods

Optimization methods that are now available that use human feedback to fine-tune LDM can be basically divided into two categories, one is acquiring new datasets [ 61 , 13 ] , and the other is changing the coefficients of loss function [ 23 ] . We have selected three methods as baseline models. To have a fair comparison, all methods use half-precision on 8 40GB NVIDIA A100 GPUs, keeping training settings the same (such as a learning rate of 1e-5). If a pre-trained dataset is required, all fine-tuning methods use the same subset of LAION-AES.

Dataset Filtering. [ 61 ] use a reward model to filter the dataset. Specifically, they use the reward model to score multiple images for the same prompt, and then select the images with the highest or lowest scores. Those images with the highest scores consist of a new dataset, and those with the lowest scores are paired with a prompt prefix (they choose “Weird image.”) to indicate that the image is relatively non-preferred. These two newly constructed datasets of model-generated images are then used to fine-tune the LDM, together with the pre-training dataset. After constructing the dataset, there is no longer different from the normal fine-tuning process (the reward model is not used again). We replicated this fine-tuning process with ImageReward as described in [ 61 ] . Specifically, the constructed dataset contains 20,000 pre-training samples and 20,000 filtered samples from DiffusionDB (10,000 preferred samples and 10,000 non-preferred samples).

Reward Weighted. [ 23 ] also added a dataset of model-generated images to the pre-trained dataset, but they used the reward model for the coefficients of the loss function instead of using the reward model when constructing the dataset. Specifically, their loss function is shown in 4 . ℒ ⁡ ( θ ) = 𝔼 ( x , z ) ∼ 𝒟 m ​ o ​ d ​ e ​ l ​ [ − r ϕ ​ ( x , z ) ​ l ​ o ​ g ​ ( p θ ​ ( x | z ) ) ] + β ​ 𝔼 ( x , z ) ∼ 𝒟 p ​ r ​ e ​ [ − l ​ o ​ g ​ ( p θ ​ ( x | z ) ) ] \mathcal{L}(\theta)=\mathbb{E}_{(x,z)\sim\mathcal{D}^{model}}[-r_{\phi}(x,z)log(p_{\theta}(x|z))]+\beta\mathbb{E}_{(x,z)\sim\mathcal{D}^{pre}}[-log(p_{\theta}(x|z))] (4) where θ \theta denotes parameters of pre-trained LDM, ϕ \phi denotes parameters of the reward model, and β \beta is a penalty parameter. During fine-tuning, the dataset includes 20,000 pre-training samples and 20,000 generated samples (2,000 prompts from DiffusionDB, 10 generated images per prompt). In [ 23 ] , the reward is constrained within the range [ 0 , 1 ] [0,1] , while our model follows an approximately normal distribution with a mean of 0 and a variance of 1. Therefore, in order to replicate their methodology, we need to map the scores to the range [ 0 , 1 ] [0,1] . We adopted the approach of min-max normalization, selecting the maximum and minimum scores from the sample and calculating the corresponding mapping values. Additionally, we set the penalty coefficient β = 0.5 \beta=0.5 .

RAFT. [ 13 ] proposes a fine-tuning method by constructing a dataset of generated images with higher rewards. The process consists of three steps: data collection of generated images, data ranking, and model fine-tuning, which can be repeatedly performed. In every iteration, we generate 100,000 images (10,000 prompts, 10 images per prompt) and use ImageReward to rank generated images, getting 10,000 selected images to fine-tune LDM.

### Appendix F More Results of ReFL

#### F.1 Demonstrations of ReFL

Figure 19 shows the insight of ImageReward score during denoising.

#### F.2 ReFL compares other fine-tuning methods

Figure 20 shows the comparison between ReFL and other fine-tuning methods. ReFL gets the highest win rate compared to any other method.

More qualitative examples of ReFL are provided in Figure 21 - 22 .

### Appendix G Additional Results of ImageReward Compared to Other Typical Image Scorers

More qualitative examples of ImageReward can be seen in Figure 23 - 26 .

### Appendix H Limitations

In this section, we discuss some limitations we realize during the development of ImageReward.

Annotation scale, diversity, and quality. Although our annotation data has reached up to about 9k prompts and 137k pairs of expert comparisons, the larger scale of the annotation dataset is still needed for better RM training. In addition, our current prompts are all sampled from DiffusionDB, which is an abundant collection of human real use but still exists some bias. Despite these prompts may close to many real cases, biases exist since the real application when people use the text-to-image model are far beyond trying strange prompts. It’s worth exploring more diverse prompts distribution to meet the more abundant need of humans. Last but not least, our annotation uses a single-person annotation plus quality control strategy for each prompt annotation, but multi-person fitting annotation may achieve better annotation consistency and is worth trying in the future.

RM training techniques. As we mentioned in Section 2.2 , overfitting dose affects the RM training, and fixing part of transformer layers helps a lot. Nevertheless, we speculate that more advanced techniques (e.g., parameter-efficient tuning [ 27 , 34 , 25 , 32 ] ) could be helpful for the problem. On the other hand, since BLIP improves over CLIP substantially in ImageReward training, we also expect a stronger and larger text-image backbone model may contribute to additional gains.