Gradient Regularization Mitigates Reward Hacking in Reinforcement Learning from Human Feedback and Verifiable Rewards — part 3/6

Cleaned-text code points [30184, 46164). Bibliography omitted here; source.md includes it.

Reference Resets are an indirect way of regularizing the gradient norm, require many more gradient steps, and do not provide a direct, controllable way to trade-off reward maximization, adherence to the initial policy, and gradient regularization. We therefore propose a novel application of explicit GR methods to RLHF and RLVR, specifically finite-difference GR ( Karakida et al., 2023 ) . To improve training stability, we implement parameter perturbations only on the transformer blocks, leaving the embedding layer and output head untouched, and clip the intermediate gradients. To make GR training efficient, we reuse the actions a ∼ π ϕ ​ ( a | s ) a\sim\pi_{\phi}(a|s) to calculate both the gradients ∇ J ​ ( ϕ , θ ) \nabla J(\phi,\theta) and ∇ J ( ϕ + ε ∇ J ( ϕ , θ ) , θ ) \nabla J(\phi+\varepsilon\nabla J(\phi,\theta),\theta) . In principle, this would require new actions a ∼ π ϕ + ε ∇ J ( ϕ , θ ) ( a | s ) a\sim\pi_{\phi+\varepsilon\nabla J(\phi,\theta)}(a|s) or correction by importance sampling. However, we empirically found this to be unnecessary and reusing actions reduces computation overhead. 1 1 1 In our MATH reasoning experiments on 8 GH200 GPUs, generating the actions took on average 7.4s, while the policy update took 60ms without GR and 150ms with GR. PyTorch-style pseudocode, implementation details and experimental details are shown in Appendix C . Our implementation is available at https://github.com/JohannesAck/gradientregularization_trl .

### 5.1 GR Mitigates Hacking Reward Models in RLHF

We first investigate whether GR can fully replace the standard KL penalty in RLHF. Scaling up from TL;DR, we run experiments on the AlpacaFarm dataset ( Dubois et al., 2023 ) , with preference feedback from GPT 4.1-Nano. We again evaluate with winrate: generating completions on the test set and judging them against reference completions with GPT4.1-Nano. We train models from the Qwen 2.5 family, running for 1000 gradient steps in the 1.5B experiments and 500 gradient steps in the 0.5B and 3B experiments and early-stop based on the training winrate. We evaluate and ablate GR in four settings: 1) no KL penalty, no GR 2) KL penalty, 3) Reference Resets + KL, 4) GR without KL. We do a grid-search to find the optimal strength for each regularization method for each model size and show results in Table 2 . In most cases using RL without regularization only yields a modest improvement over the initial SFT policy. RL with a KL penalty works decently well though Reference Resets is sometimes better. But explicit GR consistently performs best, demonstrating that it can replace the KL penalty and improve overall performance.

#### Explicit GR is robust to RM accuracy

RLHF performance can vary heavily depending on the accuracy of the RM and performance of the initial policy ( Huang et al., 2024 ) . To demonstrate robustness, we rerun our whole pipeline two more times for the 1.5B model: initial SFT, dataset sampling, RM training, GRPO with hyper-parameter tuning. In Figure 5 we show the winrate of each trained model against the accuracy of the RM with which it trained. We observe that GR performs significantly better than Reference Resets or a KL penalty when the RM is weaker, demonstrating better robustness. Reference Resets do perform slightly better than GR with the strongest RM, where reward-hacking is less prevalent.

We also find that GR is robust to choices of hyper-parameters γ \gamma and ε \varepsilon , and show a learning rate sweep in Appendix D.1 . Finally, we discover that strong GR can even compensate in robustness for a weak RM by training in a regime where the RM is more accurate than on its training distribution, see Appendix D.2 .

### 5.2 GR Prevents Focus on Easy Rule-Based Rewards

Recent work in RLVR has generally removed the KL penalty to allow for a stronger deviation from the base model ( GLM-4.5 Team et al., 2025 ; Olmo Team et al., 2025 ) , though others have kept it for training stability ( Kimi Team et al., 2025 ) . As GR does not constrain the divergence from the base model, but may provide the desired training stability, we investigate whether it can be used in RLVR to enable more flexibility while preventing reward hacking.

We perform experiments with Qwen 2.5 0.5B-Instruct and 1.5B-Instruct on GSM8K ( Cobbe et al., 2021 ) with the standard combination of formatting and correctness reward, see full details in Appendix C . We indeed observe more stable training and improved final accuracy, as shown in Figure 6 (left) and Table 3 . Notably, improved performance comes at the expense of a slightly worse adherence to the formatting reward. In the presence of both rewards, we can see the excessive focus on the easier formatting reward as a kind of reward hacking, even when neither reward is hackable on its own. This demonstrates how GR can be effective in situations with a combination of rule-based rewards.

Next, we show that a form of reward hacking is also possible even within a single reward and how GR can mitigate it. We train a Qwen 2.5-1.5B-Instruct model with a rule-based reward on MATH ( Hendrycks et al., 2021 ) . The base model achieves 46.3% pass@1 accuracy and GRPO+GR (57.6%) clearly outperforms standard GRPO (54.8%). In order to discern reward hacking, we investigate the accuracy more closely. We divide performance into quintiles of difficulty so that test-set questions are split based on the initial accuracy of their category and level-labels, as provided by the dataset. As shown in Table 4 , without regularization the performance on the easiest quintile of questions continues to improve past 250 steps. But after the first 250 steps, performance on harder questions actually degrades. This demonstrates how RL can focus on learning only the easiest questions in order to hack even a rule-based reward. In contrast, training with GR more evenly improves the accuracies across difficulties, avoiding the focus on the easier questions.

### 5.3 GR Mitigates Hacking LLM-as-a-Judge in RLVR

Finally, we investigate RL with LLM-as-a-Judge. As LLM-as-a-Judge can be susceptible to adversarial attacks ( Zhao et al., 2025 ) , we presume it can also be reward-hacked as a PR. To investigate, we re-run the previous GSM8K experiments but replace the rule-based correctness reward with an LLM judge based on Qwen2.5 1.5B-Instruct ( Qwen et al., 2025 ) . The judge receives the problem description, true answer, model response, model reasoning, and is instructed to output a score for correctness (1 to 5) which acts as the reward. As shown in Figure 7 , without regularization the model quickly starts to “hack” the judge LLM. LLM judge score goes sharply up while pass@1 accuracy on the test set peaks quite early. Empirically, we observed the model outputting excessive brackets and new HTML tags to fool the judge. In contrast, both GR and KL show much more reasonable train rewards and prevent excessive reward hacking, with GR resulting in a better final performance. Confirming results in Section 4 , we also see that, without regularization, an increase in gradient norm occurs when reward hacking begins, shown in Figure 8 (left). In Figure 9 we use different LLM judges, showing that with GR we can use a weaker judge to obtain the same result as a stronger judge without regularization. We provide additional ablations, experiments, and hyper-parameter sweeps in Appendix D .

#### Limitations

We would like to mention two limitations of our experiments and approach. While GR is empirically effective in the reasoning tasks, the assumption of a Lipschitz-continuous true reward highly depends on the chosen distance between actions. While we believe it is reasonable to consider distances in a “semantic space” as illustrated in Figure 1 (left), it could also be argued that the true reward in reasoning should be nonzero only for a unique true answer with specific formatting, violating continuity.

Another issue is that GR may inadvertently favor flat but incorrect maxima of the PR. If the PR has such maxima, using GR could even cause reward hacking and monitoring the gradient norm during training could be misleading. We did not observe this issue in our experiments and are not aware of PRs with this characteristic, but this possibility should be considered when using GR in a new setting.

## 6 Conclusion

We have investigated the problem of RL with proxy rewards (PR) and proposed a novel perspective: learning a policy in a regime where the PR is accurate. For this purpose, we have derived a theoretical connection between the flatness of an optimum and the Bradley-Terry loss of the PR at this optimum. By regularizing the gradient norm during training, we can bias the RL updates towards such a flat optimum. We first validated our theoretical analysis by using implicit gradient regularization (GR) via Reference Resets, showing they improve upon a KL penalty. We then proposed to use explicit GR based on an efficient implementation of a finite-difference estimate. Explicit GR allows us to mitigate reward model hacking of RMs in RLHF, reduce focus on easy rule-based rewards in RLVR, and alleviate format hacking with LLM-as-a-judge. We believe that GR is a promising candidate to completely replace KL penalties and improve training runs that currently eschew regularization.

## Acknowledgements

We would like to thank Soichiro Nishimori and Thanawat Lodkaew for helpful discussions. MS was supported by JST ASPIRE Grant Number JPMJAP25B1. TI was supported by KAKENHI Grant Number 22K17946.

## Impact Statement

We propose a method to prevent reward hacking in RL post-training of LLMs. We believe that preventing reward hacking is likely to have a beneficial impact in general. However, our theory only considers certain specific kinds of reward hacking, thus there is a risk that by overly relying on our method, users may miss other kinds of reward hacking, which should be monitored independently.

## Appendix A Extended related work

#### Reference Resets

We here provide an overview of alternative explanations for reference-resets and related methods. Noukhovitch et al. (2023) proposed Elastic Resets, a method which maintains an exponential moving average of the weights ϕ ¯ \bar{\phi} during the updates. After each R R steps, the current policy parameters are replaced with the EMA weights ϕ ¯ \bar{\phi} and the EMA weights are reinitialized to the initial policy ϕ 1 \phi^{1} . Next, Gorbatovski et al. (2025) proposed Trust-Region Direct Preference Optimization (TR-DPO), which proposes a method very similar to reference resets, applied to DPO ( Rafailov et al., 2023 ) . Similar to Reference Resets, the reference policy is set to the current policy after each R R updates. While DPO is meant to increase the probability of preferred responses π ⁡ ( a w | s ) \pi(a_{\mathrm{w}}|s) while decreasing the probability of un-preferred responses π ⁡ ( a l | s ) \pi(a_{\mathrm{l}}|s) , in practice it often decreases both π ⁡ ( a w | s ) \pi(a_{\mathrm{w}}|s) and π ⁡ ( a l | s ) \pi(a_{\mathrm{l}}|s) , degrading performance. This issue is known as likelihood displacement ( Razin et al., 2025 ) . Gorbatovski et al. (2025) motivate the mechanism of TR-DPO by preventing likelihood displacement, however likelihood displacement does not occur in on-policy alignment methods such as GRPO, which we investigate in this work. This mechanism can thus not explain why Reference Resets improve over standard GRPO.

Liu et al. (2025a) motivate the need for Reference Resets in reasoning training due to the magnitude of the KL penalty overwhelming the magnitude of the reward after a certain number of training steps. However, if this was the reason, we could simply use a smaller KL penalty β \beta from the beginning, which we show in Figure 16 does not work. Another reasonable approach following the reasoning of ( Liu et al., 2025a ) would be to simply decrease β \beta when the reward stagnates, but we also show in the experiments ( Figure 15 ) that this does not work as well as Reference Resets. Instead, we provide a theoretical analysis explaining its success and provide experiments validating it specifically in the RLHF setting. Ackermann et al. (2025) contemporaneously with Liu et al. (2025a) proposed reference resets as an ablation called “PPO + New KL”.

#### Replacing the KL penalty in Post-training

The most common way to prevent reward hacking with proxy rewards is a KL penalty, introduced in RLHF by Stiennon et al. (2020) and RLVR by Havrilla et al. (2024) . Some recent reports about large model training, for example GLM4.5 ( GLM-4.5 Team et al., 2025 ) and Olmo3 ( Olmo Team et al., 2025 ) remove the KL penalty, while others such as Kimi K2 ( Kimi Team et al., 2025 ) still use it. Our experiments show that both in RLHF and math experiments using gradient regularization performs better than either the KL penalty or removing the KL penalty. Direct alignment methods such as Direct Preference Optimization (DPO) ( Rafailov et al., 2023 ) , Kahneman-Tversky Optimization (KTO) ( Ethayarajh et al., 2024 ) , do not use an explicit KL penalty during training, however they indirectly optimize the KL-constrained RL objective. Instead of just a KL penalty ( Ouyang et al., 2022 ) uses PPO-ptx which adds an additional behavior cloning term to the loss, on data sampled from a pretraining dataset, to prevent regressions on standard NLP benchmarks during RLHF training.

#### Gradient Regularization

While ( Karakida et al., 2023 ) performs experiments with convolutional neural networks, ( Zhao et al., 2022 ; Zhao et al., 2024 ) further apply GR to vision transformer models. Sharpness-aware minimization (SAM) ( Foret et al., 2021 ) has been shown to correspond to gradient regularization with a specific choice of hyper-parameters ( Karakida et al., 2023 ) . ( Bahri et al., 2022 ; Zhang et al., 2022 ) apply SAM to transformer pre-training in order to improve generalization. SAM has also been used in RL, particularly by Lee et al. (2023) to improve sample efficiency when training a policy for Atari games, and by ( Lee and Yoon, 2025 ) to obtain robust policies in continuous control tasks. Our theoretical analysis in part uses the argument provided ( Lee and Yoon, 2025 ) , relating parameter flatness to action robustness. Our work is, to our knowledge, the first one to investigate the relation of GR to the accuracy of proxy rewards, as well as the first work to use GR in RLHF/RLVR post-training of LMs.

## Appendix B Proofs

Our argument can be summarized as follows: Lee and Yoon (2025) showed that flatness in parameter space is related to flatness in action space, we slightly extend the argument to a maximum error of L ^ \widehat{L} in Proposition B.5 . In our case this controls the expected proxy reward inside, when assuming a Gaussian policy with fixed covariance and full row-rank Jacobian (Assumption B.2 ). Under the assumption of β \beta smoothness of the expected proxy reward (Assumption B.3 ), this provides a bound on the gradient norm of the expected proxy reward (Lemma B.6 ). We then relate the gradient norm of the expected proxy reward to the point-wise norm of the action-gradient in Lemma B.7 . Then, as in an area with a bounded gradient norm Lipschitz continuity is guaranteed, in Lemma B.8 we show that overly sharp action pairs can only occur if at least one of the actions is outside of a certain radius from the maximum. Putting it all together yields Proposition 3.3 . Finally, we show that, under a Lipschitz continuous true reward (Assumption B.4 ), sharp minima incur an excess BT error.

### B.1 Flatness in Parameter Space implies ( δ , K , ρ ) (\delta,K,\rho) -pairwise robustness

We will first show that a ℰ − L ^ \mathcal{E}-\widehat{L} flat reward implies a D − L ^ D-\widehat{L} robust policy, closely based on the proof proposed by Lee and Yoon (2025) . We also need to define a D − L ^ D-\widehat{L} action robust policy: