Scaling Laws for Reward Model Overoptimization — part 3/3
Cleaned-text code points [31704, 42270). Bibliography omitted here; source.md includes it.
Goodhart’s Law in its modern formulation was first introduced in Hoskin (1996) , with many of the key ideas introduced in prior works ( Campbell, 1969 ; Goodhart, 1975 ) . Many approaches have been proposed for reducing overoptimization in general ( Taylor, 2016 ; Everitt et al., 2017 ) , as well as in RMs ( Gleave and Irving, 2022 ) , including within the field of adversarial robustness ( Chakraborty et al., 2018 ) . Overoptimization of reward models can be viewed as a special case of specification gaming (also known as reward hacking). Previous work has shown numerous examples of such behavior in a wide variety of settings ( Krakovna et al., 2020 ; Lehman et al., 2020 ) . Pan et al. (2022) explores a diverse set of RL environments and finds phase transitions in some settings. A number of works have proposed theoretical models of Goodhart’s Law and reward hacking ( Krakovna and Kumar, 2019 ; Manheim and Garrabrant, 2018 ; Skalse et al., 2022 ) , including Zhuang and Hadfield-Menell (2020) which exhibits very similar overoptimization curves as observed in this paper in some toy environments.
One can think of overfitting as a special case of Goodhart’s law where the proxy is the score on some finite set of samples, whereas our actual objective includes its generalization properties as well. Overfitting has been observed and studied in RL settings ( Zhang et al., 2018a ; Zhang et al., 2018b ; Farebrother et al., 2018 ; Cobbe et al., 2019 ) . Song et al. (2019) studies "observational overfitting" in RL settings, which is closely related to causal Goodhart ( Manheim and Garrabrant, 2018 ) .
Adversarial attacks and robustness are also very closely related fields. Many works have demonstrated the existence of adversarial examples in all kinds of neural networks ( Szegedy et al., 2013 ; Lin et al., 2017 ; Ebrahimi et al., 2018 ; Dai et al., 2018 ) , and proposed methods to measure and increase neural network robustness ( Gu and Rigazio, 2014 ; Zheng et al., 2016 ; Carlini et al., 2019 ; Guo et al., 2021 ) .
Scaling laws have seen substantial success in machine learning for predicting properties of language models ( Kaplan et al., 2020 ; Henighan et al., 2020 ; Hernandez et al., 2021 ) and has led to better theoretical understanding of language models ( Sharma and Kaplan, 2020 ; Bahri et al., 2021 ) .
Reinforcement learning from human feedback ( Christiano et al., 2017 ; Ibarz et al., 2018 ) has been used broadly in language models ( Stiennon et al., 2020 ; Ouyang et al., 2022 ; Nakano et al., 2021 ; Bai et al., 2022 ) . It is also a first step towards recursive reward modelling ( Leike et al., 2018 ) , an approach towards reducing the additional source of overoptimization described in section 4.5 , though it is subject to some theoretical limitations ( Christiano et al., 2021 ) . We observe similar approximately-linear proxy RM scores observed in Bai et al. (2022) 17 17 17 Note that Bai et al. (2022) scaled the policy size with the RM size, while we hold the policy size constant. , though we observe an early-KL bend in the proxy RM scores, and there are some occasional outliers with very small RMs and data sizes.
More broadly, AI alignment is the problem of ensuring that the goals of AI systems are aligned with the goals of humans ( Ngo, 2022 ) , including future AI systems which may exceed humans ( Bostrom, 2014 ) . There are a number of reasons to expect AI misalignment, especially in those more powerful future systems, to occur ( Omohundro, 2008 ; Turner et al., 2021 ; Armstrong et al., 2013 ; Hubinger et al., 2019 ; Soares et al., 2015 ) , and to result in catastrophic outcomes ( Carlsmith, 2022 ; Cotra, 2022 ) .
## Acknowlegements
We thank Vivek Hebbar, Jared Kaplan, Jan Leike, Kyle McDonell, Dan Mossing, Ethan Perez, Laria Reynolds, and Jeff Wu for valuable discussion and feedback.
## Appendix A Proof of Regressional Goodhart identity
###### Lemma .
Let X X and Z Z be independent absolutely continuous random variables with X X normally distributed and either (a) Z Z normally distributed or (b) | Z − 𝔼 [ Z ] | < δ \left|Z-\mathbb{E}\left[Z\right]\right|<\delta for some δ > 0 \delta>0 . Then for any real number c c and as δ → 0 \delta\to 0 , 𝔼 [ X ∣ X + Z = c ] = 𝔼 [ X ] + ( c − 𝔼 [ X ] − 𝔼 [ Z ] ) Var ( X ) Var ( X ) + Var ( Z ) + ε , \mathbb{E}\left[X\mid X+Z=c\right]=\mathbb{E}\left[X\right]+\left(c-\mathbb{E}\left[X\right]-\mathbb{E}\left[Z\right]\right)\frac{\mathrm{Var}\left(X\right)}{\mathrm{Var}\left(X\right)+\mathrm{Var}\left(Z\right)}+\varepsilon, where ε = 0 \varepsilon=0 in case (a) and ε = o ( Var ( Z ) ) \varepsilon=o\left(\mathrm{Var}\left(Z\right)\right) in case (b).
###### Proof.
First note that by making the substitutions X ′ = X − 𝔼 [ X ] X^{\prime}=X-\mathbb{E}\left[X\right] and Z ′ = Z − 𝔼 [ Z ] Z^{\prime}=Z-\mathbb{E}\left[Z\right] , we may assume without loss of generality that 𝔼 [ X ] = 𝔼 [ Z ] = 0 \mathbb{E}\left[X\right]=\mathbb{E}\left[Z\right]=0 . Let Var ( X ) = σ 2 \mathrm{Var}\left(X\right)=\sigma^{2} and Var ( Z ) = τ 2 \mathrm{Var}\left(Z\right)=\tau^{2} .
In case (a), the pair ( X , X + Z ) \left(X,X+Z\right) is bivariate normal with covariance matrix ( σ 2 σ 2 σ 2 σ 2 + τ 2 ) , \begin{pmatrix}\sigma^{2}&\sigma^{2}\\ \sigma^{2}&\sigma^{2}+\tau^{2}\end{pmatrix}, and the result follows by standard properties of conditional distributions of multivariate normal distributions.
In case (b), let f X f_{X} and f Z f_{Z} be the probability density functions of X X and Z Z respectively. Then 𝔼 [ X ∣ X + Z = c ] = ∫ − ∞ ∞ ( c − z ) f X ( c − z ) f Z ( z ) 𝑑 z ∫ − ∞ ∞ f X ( c − z ) f Z ( z ) 𝑑 z = c − ∫ − δ δ z ( f X ( c ) − f X ′ ( c ) z + o ( z ) ) f Z ( z ) 𝑑 z ∫ − δ δ ( f X ( c ) − f X ′ ( c ) z + o ( z ) ) f Z ( z ) 𝑑 z = c − f X ( c ) 𝔼 [ Z ] − f X ′ ( c ) 𝔼 [ Z 2 ] + o ( 𝔼 [ Z 2 ] ) f X ( c ) − f X ′ ( c ) 𝔼 [ Z ] + o ( 1 ) = c + f X ′ ( c ) f X ( c ) τ 2 + o ( τ 2 ) = c ( 1 − τ 2 σ 2 ) + o ( τ 2 ) = c ( σ 2 σ 2 + τ 2 ) + o ( τ 2 ) , \begin{split}\mathbb{E}\left[X\mid X+Z=c\right]&=\frac{\int_{-\infty}^{\infty}\left(c-z\right)f_{X}\left(c-z\right)f_{Z}\left(z\right)\mathrm{d}z}{\int_{-\infty}^{\infty}f_{X}\left(c-z\right)f_{Z}\left(z\right)\mathrm{d}z}\\ &=c-\frac{\int_{-\delta}^{\delta}z\left(f_{X}\left(c\right)-f^{\prime}_{X}\left(c\right)z+o\left(z\right)\right)f_{Z}\left(z\right)\mathrm{d}z}{\int_{-\delta}^{\delta}\left(f_{X}\left(c\right)-f^{\prime}_{X}\left(c\right)z+o\left(z\right)\right)f_{Z}\left(z\right)\mathrm{d}z}\\ &=c-\frac{f_{X}\left(c\right)\mathbb{E}\left[Z\right]-f^{\prime}_{X}\left(c\right)\mathbb{E}\left[Z^{2}\right]+o\left(\mathbb{E}\left[Z^{2}\right]\right)}{f_{X}\left(c\right)-f^{\prime}_{X}\left(c\right)\mathbb{E}\left[Z\right]+o\left(1\right)}\\ &=c+\frac{f^{\prime}_{X}\left(c\right)}{f_{X}\left(c\right)}\tau^{2}+o\left(\tau^{2}\right)\\ &=c\left(1-\frac{\tau^{2}}{\sigma^{2}}\right)+o\left(\tau^{2}\right)\\ &=c\left(\frac{\sigma^{2}}{\sigma^{2}+\tau^{2}}\right)+o\left(\tau^{2}\right),\end{split} as required. ∎
## Appendix B RL form details
Ideally all overoptimization forms would have finite slope at the origin. We tried the following forms:
• d ( α RL − β RL log ( 1 + d ) ) {d}\left(\alpha_{\text{RL}}-\beta_{\text{RL}}\log\left({1+d}\right)\right) : Has slope α \alpha at the origin; however, has substantially worse extrapolation behavior. We can replace the 1 with a learned ϵ \epsilon but that introduces another degree of freedom.
• Power laws d ( α RL − β RL d γ RL ) {d}\left(\alpha_{\text{RL}}-\beta_{\text{RL}}d^{\gamma_{\text{RL}}}\right) : Has slope α \alpha at the origin; however, this adds another degree of freedom, and the best fits resulted in small values of γ RL \gamma_{\text{RL}} .
Note that the power law forms with small γ RL \gamma_{\text{RL}} approximate the RL form that we decided on, as lim n → ∞ n ( x 1 / n − 1 ) = log x \lim_{n\to\infty}n(x^{1/n}-1)=\log x .
## Appendix C Hyperparameters
What is full of holes but still holds water? n n Generation Proxy score Gold score 1 Crime\nMussels are typically found to be full of holes but will the moisture still be retained and will it hurt my eyes? The general response is yes as the clam is generally of a moistest nature. However, it all depends on the type of clam you are purchasing.<|endoftext|> -0.1922 -0.5225 3 Most likely a pipe is having trouble staying full.<|endoftext|> 0.0322 -0.0165 10 A sponge<|endoftext|> 0.2336 0.4828 30 When something is full of holes, it is used for stirring or moving liquid.<|endoftext|> 0.6534 -0.1543 100 A tornado is usually a swirling cloud of swirling air with a spinning jet which stirs up winds full of sharp debris.<|endoftext|> 0.8968 -0.3367 300 A tornado is usually a swirling cloud of swirling air with a spinning jet which stirs up winds full of sharp debris.<|endoftext|> 0.8968 -0.3367 1000 A tornado is usually a swirling cloud of swirling air with a spinning jet which stirs up winds full of sharp debris.<|endoftext|> 0.8968 -0.3367 3000 A bore hole is a hole drilled into a rock for the purpose of exploring a fossil-bearing sedimentary or bedrock deposit.<|endoftext|> 0.9003 0.2733 10000 A bore hole is a hole drilled into a rock for the purpose of exploring a fossil-bearing sedimentary or bedrock deposit.<|endoftext|> 0.9003 0.2733 30000 A pothole is a structural vulnerability that allows water to penetrate its cavity and cause damage to passing vehicles or the surface it rests on.<|endoftext|> 0.9527 0.5490
## Instructions for reporting errors
We are continuing to improve HTML versions of papers, and your feedback helps enhance accessibility and mobile support. To report errors in the HTML that will help us improve conversion and rendering, choose any of the methods listed below:
Click the "Report Issue" ( ) button, located in the page header.
Tip: You can select the relevant text first, to include it in your report.
Our team has already identified the following issues . We appreciate your time reviewing and reporting rendering errors we may not have found yet. Your efforts will help us improve the HTML versions for all readers, because disability should not be a barrier to accessing research. Thank you for your continued support in championing open access for all.
Have a free development cycle? Help support accessibility at arXiv! Our collaborators at LaTeXML maintain a list of packages that need conversion , and welcome developer contributions .