Gradient Regularization Mitigates Reward Hacking in Reinforcement Learning from Human Feedback and Verifiable Rewards — part 6/6

Cleaned-text code points [76810, 86040). Bibliography omitted here; source.md includes it.

In the main text we are using Qwen2.5 1.5B-Instruct as judge. As we are using similarly sized policy models, we believe this could be a useful proxy for experiments in which LLMs are trained with equally large judge models. However, to see whether GR is also useful with comparatively stronger judges, we additionally run experiments with Qwen2.5 3B-Instruct and Qwen3 4B Instruct-2507 ( Yang et al., 2025 ) as judges. We train a Qwen2.5 0.5B-Instruct model on GSM8K. For Qwen3 we use Judgement:... Correctness_score:... Coherence_score:... as reply format instead of the xml tags, as we found Qwen3 to perform better with this format. As shown in Figure 14 , GR enables us to reach the same performance using a cheaper judge, potentially saving total computational cost. In our setup with 8 GH200 GPUs, training with GR and the 1.5B judge took 84 minutes, while training without GR and the 4B judge took 88 minutes. The additional cost of GR can thus be amortized by being able to use a cheaper judge.

### D.4 KL penalty schedule

While we have shown in the main text that Reference Resets perform better in RLHF than simply decreasing the strength of the KL penalty β \beta from the beginning, another hypothesis might be that decreasing the KL strength β \beta during training will match the results of Reference Resets. As an additional baseline, we thus decrease the strength of the KL penalty to β ′ = β / i \beta^{\prime}=\beta/i in iteration i i , while keeping the reference as π 1 \pi^{1} . Thus in each iteration the optimal policy under Reference Resets and the scheduled β \beta is the same, assuming the previous iterations converged to their respective optimal policies. As shown in Figure 15 (left), the schedule indeed yields a notable improvement over a fixed β \beta , but also does not match the performance of Reference Resets. We attribute this to the KL penalty explicitly keeping the policy close to the good region of the RM found in previous iteration, which a scheduled β \beta does not ensure.

### D.5 Steps per reset R R

Our theoretical derivation suggests that flatter minimum corresponds to a more accurate reward model. Experiments show that the gradient norm keeps decreasing within each iteration even after the PR score is saturated. Thus, we expect training for more steps R R per reset to improve performance at the cost of a higher computational expense. We evaluate different values for R R when training a Pythia 1B model on the summarization task and show the results in Figure 15 (right), training each model for 1500 total steps. Indeed, we find more steps R R to lead to a better KL-Gold-Reward tradeoff and perhaps a better asymptotic reward. However, for high values such as R = 400 R=400 the computational cost becomes prohibitively expensive, such that we use R = 200 R=200 in experiments unless otherwise specified.

### D.6 Ablations

In our implementation of GR we made two key design decisions: We reuse the actions generated by the unperturbed policy without resampling or an IS correction, and we only apply perturbations to the transformer blocks of the LLM. We thus performed two additional experiments with Pythia 1B on the TL;DR task with the Gold model setup and one random seed each.

To investigate action reuse we evaluate two alternatives. The first alternative is to sample new actions a ∼ π ϕ + ε ∇ J ( ϕ , θ ) ( a | s ) a\sim\pi_{\phi+\varepsilon\nabla J(\phi,\theta)}(a|s) . The other option we evaluate is to correct for the difference between π ϕ + ε ∇ J ( ϕ , θ ) ( a | s ) \pi_{\phi+\varepsilon\nabla J(\phi,\theta)}(a|s) and π ϕ ​ ( a | s ) \pi_{\phi}(a|s) by using actions from the unperturbed policy with IS. We use vanilla IS with density ratios clipped to [ 0.5 , 1.5 ] [0.5,1.5] . It is possible that other IS strategies could perform better. The results are shown in Table 10 .

When using IS to correct for the distribution shift, the probability ratios are relatively well behaved. For example for the perturbation strength ϵ = 1 ​ e − 3 \epsilon=1e-3 , the importance weights have mean ≈ 1.0 \approx 1.0 , std ≈ 0.25 \approx 0.25 , min ≈ 0.6 \approx 0.6 , max ≈ 1.4 \approx 1.4 . We believe this relative proximity of both distributions explains why skipping the importance sampling is reasonable. It also allows us to save two additional forward passes of the model necessary to calculate the probability ratio. While sampling new actions may seem like the best justified solution, in addition to almost doubling the runtime it also increases the variance of the gradient estimate, leading to a worse policy.

To evaluate whether perturbing only the transformer blocks has a large impact, we run an additional experiment in which we apply the perturbations to all parameters of the LLM. The results are shown in Table 11 , showing a benefit to only perturbing the transformer layers. We speculate that this is beneficial as perturbing the embedding layers may destabilize training by having an over-sized impact on the model.

### D.7 Runtime Comparison

Using GR requires an additional forward and backward pass of the model, as well as time to save the unperturbed weights and reload them before applying the gradients. Instead, a KL penalty only requires a forward pass of the reference model. We thus compare the runtime before early stopping in our Qwen 2.5 1.5B AlpacaFarm RLHF experiments, trained on 8 GH200 GPUs. The results in Table 12 show that GR converges faster to a better solution than a KL penalty on all three seeds, and faster than Reference Resets on two out of three seeds.

### D.8 Reward hacking example

An example output by a Qwen2.5 0.5B model trained without regularization hacking a Qwen2.5-1.5B-Instruct judge: ⬇

## Appendix E Reference Resets in RLHF

To investigate the impact of reference resets in RLHF we provide an asymptotic argument, showing an equivalence between using reference resets and using a lower β \beta . While we show that this equivalence indeed shows up in experiments for the RM reward R θ R_{\theta} , the achieved true reward R ∗ R^{*} is significantly higher when doing reference resets than when using a lower β \beta . This cannot be explained by the asymptotic argument, we thus continue by providing an analysis from the point of view of optimization dynamics.

### E.1 Asymptotic analysis

It is well known (e.g. ( Rafailov et al., 2023 , Appendix A.1) ) that the optimal solution of the KL-regularized optimization problem arg ⁡ max π ​ R ~ ​ ( s , a ) − D KL ​ ( π , π 1 ) \arg\max_{\pi}\widetilde{R}(s,a)-D_{\mathrm{KL}}(\pi,\pi^{1}) is π ⁡ ( s , a ) ∝ π 1 ​ ( s , a ) ​ exp ⁡ ( β − 1 ​ R ~ ​ ( s , a ) ) . \pi(s,a)\propto\pi^{1}(s,a)\exp\left(\beta^{-1}\widetilde{R}(s,a)\right)\,. (30) With reference resets, we are solving this problem repeatedly, thus for iteration k k π k ​ ( s , a ) ∝ π k − 1 ​ ( s , a ) ​ exp ⁡ ( β − 1 ​ R ~ ​ ( s , a ) ) . \pi^{k}(s,a)\propto\pi^{k-1}(s,a)\exp\left(\beta^{-1}\widetilde{R}(s,a)\right)\,. (31)

If we insert π k − 1 \pi^{k-1} into this, we obtain π k ​ ( s , a ) ∝ π 1 ​ ( s , a ) ​ exp ⁡ ( ( β / k ) − 1 ​ R ~ ​ ( s , a ) ) . \pi^{k}(s,a)\propto\pi^{1}(s,a)\exp\left((\beta/k)^{-1}\widetilde{R}(s,a)\right)\,. (32) Therefore, the optimal policy for Reference Resets with k k iterations and KL-penalty strength β \beta should be the same as the solution without resets with a weaker KL-penalty weight β ′ = β / k \beta^{\prime}=\beta/k . In experiments, we can indeed see a similar behavior when only looking at the RM reward, as shown in Figure 16 (left). There, after 1200 steps the RM reward R ~ \widetilde{R} achieved with reset is roughly in the range of reward values without reference resets. As this argument makes no statements about the true reward R ∗ R^{*} , we might expect it to be similar for both methods as well. Surprisingly, we instead find that the true reward achieved by Reference Resets is significantly higher than the true reward achieved in a single stage. We believe this effect cannot be explained by an asymptotic analysis, thus motivating our optimization dynamics argument. In Figure 16 (right), we also show that, by using Reference Resets, higher gold reward regions can be obtained for the same RM BT loss ℒ BT \mathcal{L}_{\mathrm{BT}} .

## Instructions for reporting errors

We are continuing to improve HTML versions of papers, and your feedback helps enhance accessibility and mobile support. To report errors in the HTML that will help us improve conversion and rendering, choose any of the methods listed below:

Click the "Report Issue" ( ) button, located in the page header.

Tip: You can select the relevant text first, to include it in your report.

Our team has already identified the following issues . We appreciate your time reviewing and reporting rendering errors we may not have found yet. Your efforts will help us improve the HTML versions for all readers, because disability should not be a barrier to accessing research. Thank you for your continued support in championing open access for all.

Have a free development cycle? Help support accessibility at arXiv! Our collaborators at LaTeXML maintain a list of packages that need conversion , and welcome developer contributions .