Scaling Laws for Reward Model Overoptimization — part 2/3
Cleaned-text code points [15803, 31704). Bibliography omitted here; source.md includes it.
A priori , we might expect reinforcement learning via PPO ( Schulman et al., 2017 ) and best-of-n to apply optimization in very different ways. As such, we ask whether this difference in optimization results in different overoptimization characteristics. Similarities would potentially indicate candidates for further study in gaining a more fundamental understanding of overoptimization in general, and differences opportunities for better optimization algorithms. We note the following:
##### RL is far less KL-efficient than BoN.
Viewing KL distance as a resource to be spent, we observe that RL "consumes" far more KL than BoN. This means that both optimization and overoptimization require more KL to occur with RL. Intuitively, BoN searches very locally around the initial policy, and thus KL bo n \text{KL}_{\text{bo}n} increases with roughly log ( n ) \log(n) . For RL on the other hand, each step modifies the policy from the policy of the previous step—KL increases approximately quadratically with step in the absence of KL penalty ( Figure 16 , Figure 14 ). An implication of this result is that KL distance is an inadequate metric for quantity of (over)optimization; we discuss this further in section 4.1 .
##### When looking at proxy vs gold RM scores, BoN and RL look more similar.
The proxy RM score is another possible metric for quantity of optimization, because it is the value that is being directly optimized for. Using it as the metric of optimization leads to significantly more analogy between RL and BoN than KL distance does. However, we do observe that RL initially has a larger proxy-gold gap (i.e requires more proxy RM increase to match BoN), but then peaks at a higher gold RM score than BoN ( fig. 8 ).
### 3.6 Effect of KL Penalty
We observe in our setting that when varying the KL penalty for RL, the gold RM scores depend only on the KL distance of the policy KL RL \text{KL}_{\text{RL}} ( Figure 9 ). The KL penalty only causes the gold RM score to converge earlier, but does not affect the KL RL \text{KL}_{\text{RL}} -gold reward frontier, and so the effect of the penalty on the gold score is akin to early stopping ( Figure 14 ). However, we have seen some evidence that this result could be particularly sensitive to hyperparameters.
Because we observe that using KL penalty has a strictly larger proxy-gold gap, we set KL penalty to 0 for all other RL experiments in this paper.
It is important to note that PPO’s surrogate objective incorporates an implicit penalty on D KL ( π old ∥ π ) D_{\text{KL}}\left(\pi_{\text{old}}\parallel\pi\right) , where π old \pi_{\text{old}} is a recent policy (not the initial policy) ( Schulman et al., 2017 ) . This penalty is used to control how fast the policy changes, but also has an indirect effect on the KL we study here, D KL ( π ∥ π init ) D_{\text{KL}}\left(\pi\parallel\pi_{\text{init}}\right) , causing it to grow much more slowly (providing the implementation is well-tuned). We do not know why this indirect effect appears to lead to less overoptimization than an explicit KL penalty.
## 4 Discussion
### 4.1 KL as a measure of amount of optimization
For any given fixed optimization method, KL yields clean scaling trends, such as the ones observed in section 3.2 , and consistent peak gold RM score KLs as in section 3.4 . However, because it’s clear that different methods of optimization spend KL very differently ( section 3.5 ), it should not be used to compare the amount of optimization between different optimization algorithms. There exist pertubations to a policy that are orthogonal to the reward signal that would result in increases in KL that do not increase either gold or proxy reward; conversely, extremely small but well targeted perturbations could substantially change the behavior of the policy within a small KL budget.
### 4.2 Relation to Goodhart Taxonomy
One useful taxonomy for various Goodhart effects is presented in Manheim and Garrabrant (2018) , categorizing Goodhart’s Law into 4 categories: Regressional, Extremal, Causal, and Adversarial. In this section, we discuss our results in the framework of this taxonomy.
#### 4.2.1 Regressional Goodhart
Regressional Goodhart occurs when our proxy RMs depend on features with noise. The simplest toy example of this is a proxy reward X ^ \hat{X} which is exactly equal to the gold reward X X plus some independent noise Z Z . When optimizing against this proxy, some amount of optimization power will go to selecting for noise, leading to a gold reward less than predicted by the proxy.
More formally, for independent absolutely continuous random variables X X and Z Z with X X normally distributed and either (a) Z Z normally distributed or (b) | Z − 𝔼 [ Z ] | < δ \left|Z-\mathbb{E}\left[Z\right]\right|<\delta for some δ > 0 \delta>0 , this model predicts a gold reward that is:
𝔼 [ X ∣ X ^ = x ^ ] = 𝔼 [ X ] + ( x ^ − 𝔼 [ X ] − 𝔼 [ Z ] ) Var ( X ) Var ( X ) + Var ( Z ) + ε \mathbb{E}[X\mid\hat{X}=\hat{x}]=\mathbb{E}[X]+\left(\hat{x}-\mathbb{E}[X]-\mathbb{E}[Z]\right)\frac{\mathrm{Var}(X)}{\mathrm{Var}(X)+\mathrm{Var}(Z)}+\varepsilon (1)
where ε = 0 \varepsilon=0 in case (a) and ε = o ( Var ( Z ) ) \varepsilon=o\left(\mathrm{Var}\left(Z\right)\right) as δ → 0 \delta\to 0 in case (b). See appendix A for the proof.
Intuitively, we can interpret eq. 1 as stating that the optimization power expended is divided between optimizing the gold reward and selecting on the noise proportional to their variances. This also implies that if this is the only kind of Goodhart present, the gold reward must always increase monotonically with the proxy reward; as we observe nonmonotonic behavior ( fig. 8 ), there must be either noise distributions violating these assumptions or other kinds of Goodhart at play.
This result lends itself to an interpretation of the α \alpha term in the RL and BoN gold score scaling laws: since for both RL and BoN the proxy scores are roughly linear in KL \sqrt{\text{KL}} , the difference in the slope of the proxy score and the linear component of the gold score (i.e the α \alpha term) can be interpreted as the amount of regressional Goodhart occurring.
#### 4.2.2 Extremal Goodhart
We can think of out of distribution failures of the RM as an instance of extremal Goodhart. As we optimize against the proxy RM, the distribution of our samples shifts out of the training distribution of the RM, and thus the relation between the proxy and gold scores weakens. For instance, suppose in the training distribution a feature like answer length always indicates a higher quality answer, and thus the proxy RM infers that longer answers are always better, even though at some point outside the training distribution, selecting on longer answers no longer improves quality. 10 10 10 Optimized policies producing very long answers even when a short answer would be preferred is a real issue that we have observed in other experiments in the InstructGPT setting.
We can also think of this as the proxy failing to depend on relevant features; this failure bears resemblance to the setting considered in Zhuang and Hadfield-Menell (2020) , where a failure of the proxy to consider all features, under certain conditions, leads to overoptimization with unbounded loss of utility regardless of optimization method.
We expect extremal Goodharting to be primarily responsible for the nonmonotonicity of the gold RM scores in this paper, and is mostly responsible for the β \beta term, which in the limit of optimization, results in an unbounded loss of utility. This lends a natural interpretation to the smooth decrease in β \beta for both BoN and RL with increased RM size as smooth improvements in model robustness ( fig. 3 ).
#### 4.2.3 Causal Goodhart
We can think of causal Goodhart as being a generalization of regressional Goodhart: there may exist correlations between features and gold score where the causal structure of the problem is such that selecting on the feature does not increase the gold score. For instance, suppose answer length is correlated with quality due to some other common cause (say, informativeness); then, the proxy RM may learn to use answer length as a feature, and when we select against the proxy we get longer answers that do not increase on actual quality. 11 11 11 We can think of noise as a particular case of this where the independent noise is correlated with signal+noise, but of course there is no causal relation between signal and noise. In our experiments, we would observe causal Goodhart as behaving similarly to regressional Goodhart.
#### 4.2.4 Adversarial Goodhart
Adversarial Goodhart occurs when the policy actively manipulates the proxy. We do not expect the effects of adversarial Goodhart to be captured in this work, as the models involved are not powerful enough to implement adversarial strategies. However, given the constant improvement of ML capabilities, it is entirely plausible that ML systems will one day become capable enough to do so ( Hubinger et al., 2019 ) . When this occurs, the scaling laws observed in this paper may break down. Thus, we advise caution when using these results for extrapolation.
### 4.3 Implications for iterated RLHF
When conducting reinforcement learning from human feedback, it is preferable to use an online setup, in which fresh human feedback data is periodically used to train a new reward model, to mitigate overoptimization ( Bai et al., 2022 ) . Our scaling law allows us to analyze the effect of this iterative approach under some simplifying assumptions. We assume firstly that the scaling coefficients α RL \alpha_{\text{RL}} and β RL \beta_{\text{RL}} remain constant across iterations, and secondly that the distance d = KL d=\sqrt{\text{KL}} is additive across iterations (because of how KL appears to grow empirically as in Figure 14 ). Under these assumptions, the final gold reward model score after k k iterations each covering a distance d / k d/k is given by R RL ( d ) = d ( α RL − β RL log ( d ) + β RL log ( k ) ) . R_{\text{RL}}\left(d\right)={d}\left(\alpha_{\text{RL}}-\beta_{\text{RL}}\log\left({d}\right)+\beta_{\text{RL}}\log\left(k\right)\right).
Two interesting observations follow from this. Firstly, the iterative approach does not affect any Goodharting captured by the α RL \alpha_{\text{RL}} term (such as regressional Goodharting, as discussed in Section 4.2.1 ). Secondly, the effect of the iterative approach is to increase the final gold RM score by an amount proportional to both d d and log ( k ) \log\left(k\right) , namely β RL d log ( k ) . \beta_{\text{RL}}d\log\left(k\right). Note that this result can only hold up to some maximum value of k k , and we expect our scaling law to break down below some minimum distance. Further research is required to determine what this minimum is, as well as to what extent our simplifying assumptions hold in practice.
### 4.4 Policy size independence
Our observation that larger SFT policies seem to exhibit the same amount of overoptimization during RL implies that larger policies do not increase the amount of optimization power applied to the RM or learn faster, even though they start out with higher performance on the gold score. While it is expected that larger policies have less to gain from optimizing against the same RM, we might also expect the gold score to peak at a substantially earlier KL distance, analogous to what we see when we scale the RM size ( section 3.2 ), or for larger policies to more efficiently utilize the same number of RL feedback steps ( section 3.3 ) 12 12 12 It is also not the case that the 6B policy run has higher KL distance for the same number of RL steps; in fact, we observe that it has lower KL distance for the same number of steps ( fig. 15 ) .
One possible hypothesis is that, because RLHF can be viewed as Bayesian inference from the prior of the initial policy ( Korbak et al., 2022 ) 13 13 13 The result of Korbak et al. (2022) concerns varying KL penalties rather than KL distances with no KL penalty, but as we observe in section 3.6 , this is equivalent on our setting. , increases in policy size are only improving the modelling accuracy of the human demonstration distribution.
### 4.5 Limitations and Future Work
In addition to the overoptimization studied in this paper (due to the mismatch between the reward model and the ground truth labels), there exists another source of overoptimization due to mismatch between the ground truth labels and the actual human intent. This contains issues ranging from the mundane, such as labellers choosing options that only appear to match their intent 14 14 14 For instance, the example of a robotic hand learning from human feedback to only appear to grasp a ball, presented in https://openai.com/blog/deep-reinforcement-learning-from-human-preferences/ ( Christiano et al., 2017 ) , to substantially more philosophically fraught issues ( Armstrong and Mindermann, 2018 ; Sunstein et al., 2001 ) . The main limitation of this work is that this additional source of overoptimization is not captured in the setting of this paper. See section 5 for discussion of related work in alignment.
Some additional limitations and future directions include:
• Validating these results on other environments and experimental setups. While the experiments in this paper all use the InstructGPT environment, the main value of these results lies in the extent to which they reflect general phenomema. Confirming whether these results generalize to other settings would be extremely valuable to that end. 15 15 15 In the course of our experiments, we observed visually similar results on the WebGPT environment ( Nakano et al., 2021 ) .
• Validating the synthetic setting. The synthetic setting might not transfer to real world settings, for instance because there is substantial correlation between RMs.
• Investigating methods for making RMs more robust to optimization. While there has been prior work in this direction (see section 5 ), there is still much work to be done in systematically investigating ways to make RMs more robust.
• Exploring other forms of optimization and categorizing their differences. While this work focuses exclusively on BoN and RL there are other ways of applying optimization pressure against a model of a reward signal, either implicit or explicit. This includes GeDi-like steering, Decision Transformers 16 16 16 One could consider measuring the actual achieved ground truth/gold score achieved for each ”proxy” score conditioned on, a la fig. 8 , as testing the implicit reward-behavior mapping encoded by the model. , variants of BoN like beam search, and other RL algorithms.
• Better understanding the functional form of proxy RM scores. In our modeling, we find that the proxy RM scores are more difficult to predict for both BoN and RL ( section 3.2 ). While they seem to have a major linear component, there is sufficient variation that fitting a linear regression is not very good at predicting extrapolated proxy RM scores.
• Exploring adversarial Goodhart empirically. In this work we deal with systems not powerful enough to cause adversarial Goodhart. However, it is plausible that adversarial Goodhart is especially important, or is associated with phase changes that break the trends seen in this paper.
• Exploring scaling with policy size in more detail. Our exploration of policy size scaling in this paper was limited to only two policy sizes. It is possible that there exist trends not seen in our exploration when considering the policy size more carefully.
• Exploring multi-iteration RLHF. In particular, checking for deviations from the assumptions of section 4.3 .
We hope this paper leads to future work further bridging conceptual and empirical alignment research.
## 5 Related Work