Scaling Laws for Reward Model Overoptimization

arxiv:2210.10760

Original source · Paper JSON

Data, training, optimization · 2022-10-19

Cites (4) · Cited by (7)

Complete cleaned Markdown · Source Markdown including bibliography

Read in parts

1. Part 1
2. Part 2
3. Part 3

Contents

• Scaling Laws for Reward Model Overoptimization
• Abstract
• 1 Introduction
• 2 Methodology
• 2.1 Synthetic Data Setup
• 2.2 Recalibration
• 3 Results
• 3.1 Fitting and validating functional forms
• 3.2 Scaling with RM Parameter Count
• 3.3 Scaling with RM Data Size
• 3.4 Scaling with Policy Size
• Larger policies see less benefit from optimization against an RM, but don’t overoptimize more.
• 3.5 RL vs BoN
• RL is far less KL-efficient than BoN.
• When looking at proxy vs gold RM scores, BoN and RL look more similar.
• 3.6 Effect of KL Penalty
• 4 Discussion
• 4.1 KL as a measure of amount of optimization
• 4.2 Relation to Goodhart Taxonomy
• 4.2.1 Regressional Goodhart
• 4.2.2 Extremal Goodhart
• 4.2.3 Causal Goodhart
• 4.2.4 Adversarial Goodhart
• 4.3 Implications for iterated RLHF
• 4.4 Policy size independence
• 4.5 Limitations and Future Work
• 5 Related Work
• Acknowlegements
• Appendix A Proof of Regressional Goodhart identity
• Lemma .
• Proof.
• Appendix B RL form details
• Appendix C Hyperparameters
• Instructions for reporting errors