ImageReward: Learning and Evaluating Human Preferences for Text-to-Image Generation — part 5/5

Cleaned-text code points [63400, 66707). Bibliography omitted here; source.md includes it.

Using RM to improve generative models. Though we have proposed ReFL as an effective method to utilize human preference scorers’ feedback to optimize LDMs, it remains an approximation of original RLHF algorithms and could be improved fundamentally. It is necessary to develop corresponding unbiased and efficient feedback learning algorithms with solid theory groundings to allow better human alignment.

### Appendix I Broader Impact

The aim of this paper is to introduce human preference feedback to improve text-to-image generation, which will help image generation to better match the needs of human life and to conform to social norms. Fine-tuning the model with human feedback helps to avoid researchers from over-relying on various types of data with copyright issues for training, and can instead directly improve performance with reward model feedback. A downside is that the preferences of a single reward model are not representative of the multiplicity of human aesthetics, and we can address this by training a variety of reward models and limiting the use of individual reward models. We believe that these benefits outweigh the drawbacks.

### Appendix J Reproducibility

We have made substantial efforts to guarantee the reproducibility of our assessments. The code and detailed information for the ImageReward model and ReFL algorithm are openly accessible in our repository (Cf. Abstract). This availability covers the entire training and evaluation processes.

Training. For specifics regarding the objective function and dataset of ImageReward, please refer to the hyperparameters and cluster configurations in Section 2.2 . Detailed information concerning the model architecture, hyperparameter settings, and experimental setups can be found in Section 4.1 .

Evaluation. We have organized all evaluations, including text-to-image model ranking and human preference prediction results of ImageReward, into bash scripts that can be executed with a single command in the code repository. Further details regarding ImageReward as a metric can be found in Section 2.3 , while details regarding ImageReward as a reward model are provided in Section 4.1 .

ReFL. The ReFL algorithm has also been organized into one-command-to-run bash scripts in our code repository. Detailed insights into ReFL can be located in Section 4.2 .

## Instructions for reporting errors

We are continuing to improve HTML versions of papers, and your feedback helps enhance accessibility and mobile support. To report errors in the HTML that will help us improve conversion and rendering, choose any of the methods listed below:

Click the "Report Issue" ( ) button, located in the page header.

Tip: You can select the relevant text first, to include it in your report.

Our team has already identified the following issues . We appreciate your time reviewing and reporting rendering errors we may not have found yet. Your efforts will help us improve the HTML versions for all readers, because disability should not be a barrier to accessing research. Thank you for your continued support in championing open access for all.

Have a free development cycle? Help support accessibility at arXiv! Our collaborators at LaTeXML maintain a list of packages that need conversion , and welcome developer contributions .