ImageReward: Learning and Evaluating Human Preferences for Text-to-Image Generation — part 2/5
Cleaned-text code points [15911, 31543). Bibliography omitted here; source.md includes it.
We thus propose an algorithm to directly fine-tune LDMs by viewing the scores of an RM as human preference losses to back-propagate gradients (Cf. Algorithm 1 ) to a randomly-picked latter step t t (in our case t ∈ [ 30 , 40 ] t\in[30,40] ) in the denoising process. The reason for the random selection of t t instead of using the last step is that, if only the gradient of the last denoising step is retained, the training is proved very unstable and the results are bad. In practice, to avoid rapid overfitting and stabilize the fine-tuning, we re-weight ReFL loss and regularize with pre-training loss. The final loss form is written as ℒ r e w a r d \displaystyle\mathcal{L}_{reward} = λ 𝔼 y i ∼ 𝒴 ( ϕ ( r ( y i , g θ ( y i ) ) ) ) \displaystyle=\lambda\mathbb{E}_{y_{i}\sim\mathcal{Y}}(\phi(r(y_{i},g_{\theta}(y_{i})))) (2) ℒ p r e \displaystyle\mathcal{L}_{pre} = 𝔼 ( y i , x i ) ∼ 𝒟 ( 𝔼 ℰ ( x i ) , y i , ϵ ∼ 𝒩 ( 0 , 1 ) , t [ ‖ ϵ − ϵ θ ( z t , t , τ θ ( y i ) ) ‖ 2 2 ] ) \displaystyle=\mathbb{E}_{(y_{i},x_{i})\sim\mathcal{D}}(\mathbb{E}_{\mathcal{E}(x_{i}),y_{i},\epsilon\sim\mathcal{N}(0,1),t}[\|\epsilon-\epsilon_{\theta}(z_{t},t,\tau_{\theta}(y_{i}))\|_{2}^{2}]) (3) where θ \theta denotes the parameters of the LDM, g θ ( y i ) g_{\theta}(y_{i}) denotes the generated image of LDM with parameters θ \theta corresponding to prompt y i y_{i} . Meanings of other symbols are detailed in Algorithm 1 , while the loss function of ℒ p r e \mathcal{L}_{pre} is taken from [ 45 ] .
### 4 Experiment
#### 4.1 ImageReward: On Human Preference Prediction
Dataset & Training Setting. Rankings of annotated images are collected to train ImageReward, which contains 8,878 prompts and 136,892 pairs of image comparisons. We divide the dataset according to prompts annotated by different annotators and select 466 prompts from annotators who have a higher agreement with researchers to consist for the model test. Except for prompts for testing, other more than 8k prompts of annotation are collected for training.
We load the pre-trained checkpoint of BLIP (ViT-L for image encoder, 12-layers transformer for text encoder) as the backbone of ImageReward, and initialize MLP head according to 𝒩 ( 0 , 1 / ( d m o d e l + 1 ) ) \mathcal{N}(0,1/(d_{model}+1)) decaying the learning rate with a cosine schedule. We sweep over several value settings of learning rate and batch size and fix different rates of backbone transformer layers. We find that fixing 70% of transformer layers with a learning rate of 1e-5 and batch size of 64 can reach up to the best preference accuracy. ImageReward is trained on 4 40GB NVIDIA A100 GPUs, with a per-GPU batch size of 16.
We use the CLIP score, Aesthetic score, and BLIP score as baselines to compare with the ImageReward. CLIP score and BLIP score are calculated directly as cosine similarity between text and image embedding, while the Aesthetic score is given by an aesthetic predictor introduced by LAION [ 50 ] .
Agreement Analysis. Agreement assesses the likelihood of two individuals sharing consistent preferences for superior images. While most people generally agree on image quality, variations in model-generated images may lead to divergent judgments. Before assessing model performance, it’s crucial to measure the likelihood of consensus in selecting superior images. We use other 40 prompts (778 pairs) to calculate preference agreement between researchers, annotators, annotator ensemble, and models. Table 2(a) shows the result.
Main Results: Preference Accuracy. Preference accuracy is the correctness of a scorer choosing the same one from two different images of one prompt with a human. As Table 3 shows, our model outperforms all the baselines. The preference accuracy of ImageReward reaches up to 65.14%, which is 15.14% more than 50% (random), about twice as much as 7.76% (that of BLIP score).
Main Results: Human Evaluation. To evaluate the ability of ImageReward to select the more preferred images among large amounts of generated images, we produce another dataset, collecting prompts with 9/25/64 generated images from DiffusionDB, and use different methods to select from those images to get top3 results. Then three annotators rank these selected top-3 images. Figure 5 shows the win rates. Qualitative results can be seen in Appendix G , showing that ImageReward can select images that are more aligned to text and with higher fidelity and avoid toxic contents.
Ablation Study: Training dataset size. To investigate the effect of training dataset sizes on the performance of the model, comparative experiments are conducted. Table 2(b) shows that adding up the scale of the dataset significantly improves the preference accuracy of ImageReward. It’s promising that if we collect more annotation data in the future, ImageReward will get better performance.
Ablation Study: RM backbone. ImageReward adopts BLIP as the backbone, which may raise curiosity about how well BLIP compares to CLIP. We add MLP to CLIP, training in a similar way, and the result is also shown in Table 2(b) . Even if CLIP uses a relatively larger training data set, its preference is still inferior to that of BLIP. The difference between these two as backbone may partly be because BLIP used bootstrapping of its training set. Moreover, we use BLIP’s image-grounded text encoder as a feature encoder different from the separate encoder for text/image as CLIP.
#### 4.2 ReFL: On Improving Diffusion Models with Human Preference
Training Settings. We use Stable Diffusion v1.4 [ 45 ] as the baseline generative model and fine-tune it for experiments. For the dataset, the pre-training dataset is from a 625k subset of LAION-5B [ 50 ] selected by aesthetic score, while the prompt set for ReFL is sampled from DiffusionDB. The model is fine-tuned in half-precision on 8 40GB NVIDIA A100 GPUs, with a learning rate of 1e-5 and batch size of 128 in total (64 for pre-training and 64 for ReFL). For ReFL algorithm, we set ϕ = R e L U , λ = 1 e − 3 \phi=ReLU,\lambda=1e-3 and T = 40 , [ T 1 , T 2 ] = [ 1 , 10 ] T=40,[T_{1},T_{2}]=[1,10] .
Evaluation Settings. We collect 466 real user prompts from DiffusionDB and 90 designed challenging prompts from multi-task benchmark (MT Bench) [ 40 ] for evaluation. All fine-tuning methods use the same dataset as the pre-training dataset or generated dataset (both contain 20,000 samples), and train for one epoch with the same training settings (the same learning rate and batch size) for a fair comparison. The human evaluation is consistent with Section 2.3 and the form of dataset labeling, which involves humans sorting multiple images under a prompt. Table 4 and Figure 6 show the comparison results. All methods use the same pre-trained Stable Diffusion v1.4 and the same reward model ImageReward, using PNDM [ 30 ] noise scheduler and default classifier free guidance scale of 7.5 for inference.
We compare several important related methods for improving text-to-image generation [ 61 ; 23 ; 13 ] , whose implementation details are provided in Appendix E . Compared to ReFL’s direct tuning, these previous methods are all based on indirect data augmentation or loss reweighing.
Results and analysis. When compared to the original version, ReFL fine-tuned model is mostly preferred with the most win rate and the highest win rate. When compared to each other, ReFL is always the preferred one.
Note that in our evaluation, neither RAFT [ 13 ] nor Reward Weighted [ 23 ] has become better compared to the baseline, although they have been verified in their own experiments. Note that both RAFT and Reward Weighted do not collect the prompts used by users in real scenarios at finetune, whereas Reward Weighted manually constructs a dataset to address the alignment issue by combining colors, numbers, backgrounds, and objects. The prompts used in our review are more widely distributed and complex, so the problems with their methods are more clearly exposed.
RAFT [ 13 ] suffers from over-fitting as the number of iterations increases. [ 13 ] propose using an expert generator as a regularizer to avoid overfitting the reward model. However, RAFT is constrained by the quality of the constructed dataset. It is important to note that even expert generators have limitations, and when fine-tuning is performed using prompts sampled from real user data, which can be challenging, there may be instances where the expert generator fails to generate high-quality images.
In the case of the Reward Weighted method [ 23 ] , although real images are used for regularization, there is a problem with the coefficient used for the rewards, which is constrained within the [0, 1] range. This implies that while preferred images are given larger weights and poor images are given smaller weights, the influence of the non-preferred images is not completely eliminated. Similarly, when utilizing real user prompts, it is likely that there will be non-preferred images (even those relatively the best) in the dataset, which can introduce interference. The failure to eliminate the impact of non-preferred images hinders the effectiveness of the Reward Weighted method.
Dataset Filtering [ 61 ] , on the other hand, considers real images and handles non-preferred images by labeling them as "Weird image." However, this influence is indirect. In contrast, our proposed algorithm provides direct gradient feedback through rewards, allowing for guidance toward a "better" direction, which enables more effective problem-solving.
In summary, RAFT is constrained by the limited ability of the generator, and the Reward Weighted method suffers from the influence of non-preferred images due to the choice of reward coefficients. Dataset Filtering partially addresses the problem by considering real images and labeling abnormal images, but it is still indirect and limited. By directly incorporating rewards into the gradient feedback, our proposed algorithm ReFL offers a more effective solution to these challenges. Qualitative examples are in Figure 7 .
### 5 Related Work
Text-to-image Generation and Evaluation. Text-to-image generation has come a long way since the popularization of GANs [ 17 ] , with key developments including models like DALL-E [ 43 ] and CogView [ 11 ] . Recently, diffusion models [ 52 ; 19 ; 10 ; 47 ] have achieved remarkable results, with Stable Diffusion [ 45 ] being particularly popular. Evaluation metrics such as Inception Score (IS) [ 4 ] and Fréchet Inception Distance (FID) [ 18 ] are commonly used to assess model performance after fine-tuning, but they cannot evaluate either single image generations or text-image coherence.
For evaluating individual generated images based on a prompt, prior works [ 42 ; 46 ; 63 ] often use CLIP [ 41 ] to calculate text-image similarity. While these metrics are useful, they don’t capture human preference comprehensively. Other predictors, like Aesthetic from LAION [ 50 ] , partially contribute to this holistic evaluation by scoring image aesthetics using a CLIP-based architecture. RM in RLHF, on the other hand, considers a mixture of elements such as text-image alignment, fidelity, and aesthetics. Overall, RM such as ImageReward provides a more complete evaluation for individual text-to-image generations, making it better aligned with human preferences.
Learning from Human Feedback. There is often a gap between generative models’ pre-training objectives and human intent. Thus human feedback has been utilized to align model performance with intent in various language applications [ 1 ; 22 ; 36 ; 67 ; 65 ] via training an RM [ 35 ; 8 ; 59 ; 20 ; 24 ] to learn human preference. Researchers have explored RL for language models to achieve more truthful, helpful, and harmless outcomes [ 39 ; 2 ; 68 ; 60 ; 55 ; 29 ; 49 ; 3 ] . Previous work [ 5 ; 55 ] used human feedback to train reward functions for summarization tasks, while InstructGPT [ 39 ] applied RLHF to GPT-3 for multi-task NLP, yielding significant improvements.
In text-to-image generation, however, there have been few studies on the topic. One concurrent work [ 23 ] has focused on text-image coherence in the closed domain using simple synthetic prompts based on templates, and propose to improve models using loss re-weighing. Other concurrent works [ 61 ; 21 ; 13 ] collects 1-of-n selection from noisy online user clicking, and thus do not enforce consistent standards and prompt diversity. Their optimization methods are based on indirect data filtering and augmentation. On the contrary, ImageReward serves as a general-purpose human preference scorer with quality ensured by rigorous annotation pipeline, and corresponding ReFL is the first direct tuning method for optimize diffusion models from scorer feedback.
### 6 Conclusion
In this work, we have presented ImageReward and ReFL, the first general-purpose text-to-image human preference reward model, and a direct fine-tuning approach for optimizing diffusion models by ImageReward feedback. Through our systematic pipeline for human preference annotation, we curate a dataset of 137k expert comparisons to train ImageReward, and build ReFL algorithm on top of it. They together address prevalent issues in generative models and help to better align text-to-image generation with human values and preferences.
### Acknowledgement
We would like to thank the data annotators for their help and support. This research was supported by the Technology and Innovation Major Project of the Ministry of Science and Technology of China under Grant 2022ZD0118600 and 2022ZD0118601, Natural Science Foundation of China (NSFC) for Distinguished Young Scholars No. 61825602, NSFC No. 62276148, a research fund from Zhipu.AI, and the New Cornerstone Science Foundation through the XPLORER PRIZE.
### Appendix A Details on ImageReward’s Comparison Data Annotation Pipeline
#### A.1 Prompt Selection
Training human preference RM requires a diverse prompt distribution that could cover and represent users’ authentic usage. DiffusionDB has 1.8M prompts which are far beyond the number we plan to annotate. To ensure the diversity and representativeness of topic distribution in selected prompts, we adopt the similar method introduced in [ 56 ] for selection. During the graph-based selection, every sample, which is prompt in our case, is represented by a vector calculated by Sentence-BERT [ 44 ] . Then a graph is constructed with represented samples as vertices and every vertex is connected to k k nearest neighbors, where k k is a hyper-parameter in the algorithm and k = 150 k=150 is found to perform well. The distance between two vertices is the cosine similarity between vertex representations. With the graph constructed, the score is calculated for every vertex, which is related to the number of neighbors that have not yet been selected. Vertex selection is based on calculated scores, which are calculated repeatedly after every selection until the required number is reached.
Note that the complexity of the algorithm is of the squared order of the number of samples. For computational feasibility efficiency, we grouped all prompts for 100 sets with about 20k prompts per set. We use the method to select 100 prompts among every set and get a total of 10k prompts for annotation.
#### A.2 Annotation Management
We cooperate with a professional annotation company to complete professional annotation. Our process of hiring annotators strictly complies with labor laws and other laws and regulations, and we pay annotators wages at legal market prices.