Gradient Regularization Mitigates Reward Hacking in Reinforcement Learning from Human Feedback and Verifiable Rewards — part 1/6
Cleaned-text code points [0, 15544). Bibliography omitted here; source.md includes it.
# Gradient Regularization Mitigates Reward Hacking in Reinforcement Learning from Human Feedback and Verifiable Rewards
###### Abstract
Reinforcement Learning from Human Feedback (RLHF) or Verifiable Rewards (RLVR) are two key steps in the post-training of modern Language Models (LMs). A common problem is reward hacking, where the policy may exploit inaccuracies of the reward and learn an unintended behavior. Most previous works address this by limiting the policy update with a Kullback-Leibler (KL) penalty towards a reference model. We propose a different framing: Train the LM in a way that biases policy updates towards regions in which the reward is more accurate. First, we derive a theoretical connection between the accuracy of a reward model and the flatness of an optimum at convergence. Gradient regularization (GR) can then be used to bias training to flatter regions and thereby maintain reward model accuracy. We confirm these results by showing that the gradient norm and reward accuracy are empirically correlated in RLHF. We then empirically show that Reference Resets of the KL penalty find flatter regions with a higher reward accuracy. We further improve on this by proposing to use explicit GR with an efficient finite-difference estimate. Empirically, GR performs better than a KL penalty across a diverse set of RL experiments with LMs. GR achieves a higher GPT-judged win-rate in RLHF, avoids overly focusing on the format in rule-based math rewards, and prevents hacking the judge in LLM-as-a-Judge math tasks.
###### Keywords:
## 1 Introduction
Reinforcement Learning (RL) has become a key part of the post-training of language models (LMs) ( Stiennon et al., 2020 ; Shao et al., 2024 ) . In the case of RL from Human Feedback (RLHF) ( Christiano et al., 2017 ) , we use RL to align the behavior of LMs with human preferences, which we cannot easily represent with a rule-based reward. In the case of RL from Verifiable Feedback (RLVR) ( Havrilla et al., 2024 ; Lambert et al., 2024 ) , RL is used to improve the performance on tasks with verifiable rewards, such as mathematical reasoning or agentic tasks. In RLHF, we use pairwise comparison data to train a reward model (RM), which then provides the reward estimates for policy updates. In RLVR, we use a verifier such as a rule-based reward or another Large Language Model (LLM) to check if the model output matches the expected answer. In both cases there is a desired behavior, corresponding to a desired true reward, which we try to approximate with the trained RM, rule-based reward, or LLM-as-a-judge reward. We collectively refer to them as proxy rewards (PRs). A key challenge of RL post-training is: How can we ensure that when updating our policy with the PR, we actually improve the true reward, i.e., how can we ensure that our PR stays accurate as the policy changes throughout training? One solution is to iteratively update the PR during training with new data from the current policy ( Christiano et al., 2017 ) . As this can be costly, another option is to use a Kullback-Leibler (KL) penalty to ensure the policy stays close to the initial model ( Stiennon et al., 2020 ) . In practice, the KL penalty slows down training and may not even improve performance ( Gao et al., 2023 ) , leading to recent papers abandoning it for tasks with rule-based rewards ( GLM-4.5 Team et al., 2025 ; Olmo Team et al., 2025 ) . However, this risks reward hacking with reward models and LLM-as-a-judge.
We thus aim to modify the policy update such that the policy not only maximizes the PR, but also maximizes the PR accuracy, without constraining it to stay close to the initial policy. We argue that reward hacking often corresponds to sharp optima of the PR and propose gradient regularization (GR) as a solution. We illustrate an overview in Figure 1 .
In Section 3 , we formalize this notion and show a theoretical connection between the flatness of an optimum and the PR accuracy at this optimum, as measured by the Bradley-Terry ( Bradley and Terry, 1952 , BT;) loss. In Section 4 , we show that this theoretical connection can be utilized to improve RLHF in practice. We leverage a recent method, Reference Resets ( Liu et al., 2025a ) , and demonstrate that it implicitly regularizes the gradient, providing a novel interpretation of this method. We empirically show this enables better RLHF performance on the TL;DR task. In Section 5 , we propose a novel application of explicit gradient regularization to RL post-training. We demonstrate that it outperforms baselines in RLHF tasks with reward models as well as rule-based and LLM-as-a-judge RLVR math tasks. By using GR, we are able to train proficient LLM-as-a-Judge and RLHF models completely replacing the standard KL penalty .
## 2 Background
### 2.1 Reinforcement Learning
As is common in the RL post-training literature ( Shao et al., 2024 ; Ahmadian et al., 2024 ) , we consider an episode consisting of a single state s s , representing the prompt, and a single action a a , representing the reply. We denote the state set by 𝒮 \mathcal{S} and the action set by 𝒜 \mathcal{A} . As policy π ϕ \pi_{\phi} with parameters ϕ \phi , we consider an autoregressive LM with conditional probability π ϕ ( a | s ) = ∏ t π ϕ ( 𝔞 t | s , 𝔞 < t ) \pi_{\phi}(a|s)=\prod_{t}\pi_{\phi}(\mathfrak{a}_{t}|s,\mathfrak{a}_{<t}) , where 𝔞 t \mathfrak{a}_{t} is the t t -th response token, 𝔞 < t \mathfrak{a}_{<t} refers to the response tokens before 𝔞 t \mathfrak{a}_{t} and a a refers to the entire response. We assume that there is a true reward function R ∗ : 𝒮 × 𝒜 → ℝ R^{*}:\mathcal{S}\times\mathcal{A}\to\mathbb{R} and our goal is to maximize the return of the policy π ϕ \pi_{\phi} on this reward J ∗ ( ϕ ) = 𝔼 s ∼ P ( s ) , a ∼ π ϕ [ R ∗ ( s , a ) ] J^{*}(\phi)=\expect_{s\sim P(s),a\sim\pi_{\phi}}[R^{*}(s,a)] . However, we do not have access to R ∗ R^{*} . Instead, we use a proxy reward (PR) R ~ \widetilde{R} , and train the policy to maximize its return J ( ϕ , θ ) = 𝔼 s ∼ P ( s ) , a ∼ π ϕ [ R ~ θ ( s , a ) ] J(\phi,\theta)=\expect_{s\sim P(s),a\sim\pi_{\phi}}[\widetilde{R}_{\theta}(s,a)] . To optimize the return we use REINFORCE policy gradient updates ( Williams, 1992 ) ∇ ϕ J ( ϕ , θ ) \displaystyle\nabla_{\phi}J(\phi,\theta) = \displaystyle= (1) 𝔼 s ∼ P ( s ) , a ∼ π ϕ \displaystyle\expect_{s\sim P(s),a\sim\pi_{\phi}} [ ( R ~ θ ( s , a ) − b ( s ) ) ∇ ϕ log π ϕ ( a | s ) ] , \displaystyle\left[(\widetilde{R}_{\theta}(s,a)-b(s))\nabla_{\phi}\log\pi_{\phi}(a|s)\right]\,, where b ( s ) b(s) is a baseline. In this work, we use a variant of Group Relative Policy Optimization (GRPO) ( Shao et al., 2024 ) called GRPO Done Right (Dr.GRPO) ( Liu et al., 2025b ) , which removes normalization terms from GRPO. As we do not do multiple updates per batch, Dr.GRPO simplifies to using REINFORCE with the baseline being the sample average over multiple actions drawn for the same state, i.e., b ( s ) = 1 N ∑ i = 1 N R ~ ( s , a i ) b(s)=\frac{1}{N}\sum_{i=1}^{N}\widetilde{R}(s,a^{i}) , with a i ∼ π ϕ ( a | s ) a^{i}\sim\pi_{\phi}(a|s) . GRPO was originally presented with a KL penalty D KL ( π ϕ , π ϕ 1 ) D_{\mathrm{KL}}(\pi_{\phi};\pi_{\phi^{1}}) , as commonly used in RLHF ( Stiennon et al., 2020 ) . This KL-penalty, weighted by a hyper-parameter β \beta , is intended to keep the model close to the initial policy π ϕ 1 \pi_{\phi^{1}} , preventing reward hacking ( Stiennon et al., 2020 ) , but is sometimes omitted in more recent works ( GLM-4.5 Team et al., 2025 ; Olmo Team et al., 2025 ) .
### 2.2 Proxy Rewards
We consider three types of PRs: Trained reward models R θ R^{\theta} , rule based rewards R R R^{\mathrm{R}} , and LLM-as-a-Judge rewards R LM R^{\mathrm{LM}} .
#### Reward Models
(RMs) are commonly used in RLHF ( Christiano et al., 2017 ; Stiennon et al., 2020 ) to represent complex preferences which are hard to turn into rule-based rewards. They generally assume the Bradley-Terry (BT) model of preference ( Bradley and Terry, 1952 ) where the probability P P of preferring one option a 1 a_{1} over another option a 0 a_{0} is the logistic function σ ( x ) = 1 / ( 1 + e − x ) {\sigma(x)=1/(1+e^{-x})} of the difference of the true reward for each action, i.e., P ( a 1 > a 0 ∣ s ) = σ ( R ∗ ( s , a 1 ) − R ∗ ( s , a 0 ) ) . P(a_{1}>a_{0}\mid s)=\sigma\left(R^{*}(s,a_{1})-R^{*}(s,a_{0})\right)\,. (2) Pairs of responses ( a 0 , a 1 ) (a_{0},a_{1}) for each prompt s s are collected from an initial model π ϕ 1 \pi_{\phi^{1}} . Human annotators then choose their preferred (winning) reply a w a_{\mathrm{w}} and not preferred (losing) reply a l a_{\mathrm{l}} , creating a dataset 𝒟 RM = { ( s j , a w j , a l j ) } j = 1 N RM \mathcal{D}_{\mathrm{RM}}=\{(s^{j},a_{\mathrm{w}}^{j},a_{\mathrm{l}}^{j})\}_{j=1}^{{N_{\mathrm{RM}}}} . We can then use the BT assumption to train an RM R θ : 𝒮 × 𝒜 → ℝ R_{\theta}:\mathcal{S}\times\mathcal{A}\to\mathbb{R} with parameters θ ∈ Θ \theta\in\Theta by minimizing the cross-entropy loss based on the BT model: ℒ BT ( θ , ϕ ) = \displaystyle\mathcal{L}_{\mathrm{BT}}(\theta,\phi)= (3) − 𝔼 ( s , a w , a l ) ∼ P π ϕ [ log σ ( R θ ( s , a w ) − R θ ( s , a l ) ) ] , \displaystyle-\expect_{(s,a_{\mathrm{w}},a_{\mathrm{l}})\sim P_{\pi_{\phi}}}[\log\sigma(R_{\theta}(s,a_{\mathrm{w}})-R_{\theta}(s,a_{\mathrm{l}}))]\,, where P π ϕ = P ( s ) π ϕ ( a 0 | s ) π ϕ ( a 1 | s ) P ( a 1 > a 0 | s ) P_{\pi_{\phi}}=P(s)\pi_{\phi}(a_{0}|s)\pi_{\phi}(a_{1}|s)P(a_{1}>a_{0}|s) is the probability of the ( s , a w , a l ) (s,a_{\mathrm{w}},a_{\mathrm{l}}) triplet under the policy π ϕ \pi_{\phi} . To train the RM we minimize ℒ BT ( θ , ϕ 1 ) \mathcal{L}_{\mathrm{BT}}(\theta,\phi^{1}) and the expectation is replaced by the sample average from 𝒟 RM \mathcal{D}_{\mathrm{RM}} .
#### Rule-Based Rewards
( Havrilla et al., 2024 ) are deterministic checks whether a final answer matches the ground truth. To encourage reasoning, and to allow a discrimination of a final answer against occurrences of the answer during reasoning, rule-based rewards typically require the answer to follow a specific format, for example using the LaTeX tag \boxed{} ( Hendrycks et al., 2021 ) or the HTML tags ( DeepSeek-AI, 2025 ) <think>...</think><answer>...</answer> . Thus, oftentimes one reward term R F R^{\mathrm{F}} checks whether the format matches and another reward term R C R^{\mathrm{C}} checks for correctness, to give the rule-based reward R R = R C + R F R^{\mathrm{R}}=R^{\mathrm{C}}+R^{\mathrm{F}} .
#### LLM-as-a-Judge
( Zheng et al., 2023 ) prompts an LLM and uses its textual output to check whether an answer is correct. Frequently designing a rule-based reward can be challenging due to many possible accurate solutions, e.g., the correctness of a proof or many equivalent ways to write a math answer. Instead, prompting an LLM-as-a-Judge with a description of scoring criteria, the question, and the correct answer (if available) can allow us to capture solutions more robustly. LLM-as-a-Judge can also allow for more complicated rewards with a combination of objective (e.g., correct reply) and subjective (e.g., clear reasoning) criteria.
### 2.3 Gradient Regularization (GR)
Flat minima of a loss function ℒ ( ϕ ) \mathcal{L}(\phi) are connected to better generalization in supervised learning ( Hochreiter and Schmidhuber, 1997 ; Foret et al., 2021 ) , i.e., a smaller difference between the population/test loss ℒ ( ϕ ) = 𝔼 ( x , y ) ∼ P ( x , y ) [ ℓ ( f ϕ ( x ) , y ) ] \mathcal{L}(\phi)=\mathbb{E}_{(x,y)\sim P(x,y)}[\ell(f_{\phi}(x),y)] and its empirical approximation on a finite training dataset 𝒟 \mathcal{D} , consisting of i.i.d. input-label pairs ( x , y ) ∼ P ( x , y ) (x,y)\sim P(x,y) , for a loss function ℓ \ell . A way to obtain flat minima is by regularizing the gradient norm of the objective ℒ \mathcal{L} , i.e., the squared Euclidean norm ‖ ∇ ϕ ℒ ( ϕ ) ‖ 2 \|\nabla_{\phi}\mathcal{L}(\phi)\|^{2} ( Zhao et al., 2022 ) . Adding this term to our loss function, we need to calculate its gradient, which can be approximated with a parameter perturbation ( Karakida et al., 2023 ) Δ ϕ ‖ ∇ ϕ ℒ ( ϕ ) ‖ 2 = ∇ ϕ ℒ ( ϕ + ε ∇ ϕ ℒ ( ϕ ) ) − ∇ ϕ ℒ ( ϕ ) ε . \Delta_{\phi}\|\nabla_{\phi}\mathcal{L}(\phi)\|^{2}=\frac{\nabla_{\phi}\mathcal{L}(\phi+\varepsilon\nabla_{\phi}\mathcal{L}(\phi))-\nabla_{\phi}\mathcal{L}(\phi)}{\varepsilon}\,. (4) The model parameters are then updated as ϕ ← ϕ − η ∇ ϕ ℒ ( ϕ ) − η γ 2 Δ ϕ ‖ ∇ ϕ ℒ ( ϕ ) ‖ 2 , \phi\leftarrow\phi-\eta\nabla_{\phi}\mathcal{L}(\phi)-\eta\frac{\gamma}{2}\Delta_{\phi}\|\nabla_{\phi}\mathcal{L}(\phi)\|^{2}\,, (5) where η ∈ ℝ + \eta\in\mathbb{R}^{+} is the learning rate and γ ∈ ℝ \gamma\in\mathbb{R} is a hyper-parameter controlling the strength of the GR and ε \varepsilon controls the strength of the parameter perturbation.
## 3 Accurate Proxy Rewards via Gradient Regularization
### 3.1 Problem Formulation and Overview
As mentioned above, the goal of RL is to learn a policy π ϕ \pi_{\phi} that maximizes the expected true reward R ∗ R^{*} . As we do not have access to R ∗ R^{*} , we instead have to use the PR R ~ \widetilde{R} to update our policy. As the PR is generally an approximation of the true reward R ∗ R^{*} , it may be prone to reward hacking and we need to ensure that the PR stays accurate during training. For PRs that have been trained or designed based on samples from an initial policy π 1 \pi^{1} , such as RMs in RLHF, the most common solution is to use a KL penalty ( Stiennon et al., 2020 ; Shao et al., 2024 ) ensuring that the policy stays close to π 1 \pi^{1} and thus the PR stays accurate. However, the KL penalty also limits how much the policy can learn.
Instead of changing the PR R ~ \widetilde{R} or constraining π ϕ \pi_{\phi} , we aim to update the policy π ϕ \pi_{\phi} such that it obtains a high reward provided by the PR R ~ \widetilde{R} , while also biasing the policy update towards regions in which the PR R ~ \widetilde{R} is accurate. Our goal is thus max ϕ 𝔼 a ∼ π ϕ [ R ~ ( s , a ) ] − γ ℒ BT ( θ , ϕ ) , \max_{\phi}\mathbb{E}_{a\sim\pi_{\phi}}[\widetilde{R}(s,a)]-\gamma\mathcal{L}_{\mathrm{BT}}(\theta,\phi)\,, (6) where γ > 0 \gamma>0 is a hyper-parameter and we use the BT loss ℒ BT ( θ , ϕ ) \mathcal{L}_{\mathrm{BT}}(\theta,\phi) of a PR parameterized with θ \theta on actions sampled from π ϕ \pi_{\phi} . However, we cannot directly evaluate the BT loss ℒ BT ( θ , ϕ ) \mathcal{L}_{\mathrm{BT}}(\theta,\phi) , as it requires pairwise comparisons for actions drawn from π ϕ \pi_{\phi} . As we will show below, under some assumptions, overly sharp optima of the objective J ( ϕ , θ ) J(\phi,\theta) correspond to overly sharp maxima of the PR R ~ \widetilde{R} , which imply an excess BT loss ℒ BT \mathcal{L}_{\mathrm{BT}} of the PR. Regularizing the policy gradient norm during training biases the optimization towards flat optima ( Zhao et al., 2022 ) , avoiding this problem. Instead of ℒ BT ( θ , ϕ ) \mathcal{L}_{\mathrm{BT}}(\theta,\phi) , we thus use the policy gradient norm ‖ ∇ ϕ J ( ϕ , θ ) ‖ 2 \|\nabla_{\phi}J(\phi,\theta)\|^{2} , yielding the practically optimizable max ϕ 𝔼 a ∼ π ϕ [ R ~ ( s , a ) ] − γ ‖ ∇ ϕ J ( ϕ , θ ) ‖ 2 . \max_{\phi}\mathbb{E}_{a\sim\pi_{\phi}}[\widetilde{R}(s,a)]-\gamma\left\|\nabla_{\phi}J(\phi,\theta)\right\|^{2}\,. (7)
### 3.2 Gradient Regularization Improves Proxy Reward Accuracy