Gradient Regularization Mitigates Reward Hacking in Reinforcement Learning from Human Feedback and Verifiable Rewards — part 2/6

Cleaned-text code points [15544, 30184). Bibliography omitted here; source.md includes it.

Our argument consists of three steps: GR biases optimization towards flat maxima. Flat maxima imply pairwise robust policies. Pairwise robust policies correspond to accurate PRs, under the assumption of a flat reward. We provide an illustration in Figure 2 . We note that our argument in this section focuses on continuous action spaces, while LMs use discrete action spaces. We discuss this discrepancy at the end of this section and provide experimental evidence in the LM setting in Section 4 and Section 5 .

#### GR Favors Flat Maxima in Parameter Space

It is well known that GR favors flat minima in the parameter space ( Barrett and Dherin, 2021 ; Karakida et al., 2023 ) . One way to see this is the argument of Zhao et al. (2022) , connecting the gradient norm to Lipschitz continuity, which we reproduce for completeness in Appendix B.3 . It is also well known in the supervised learning literature ( Hochreiter and Schmidhuber, 1997 ) that flat minima lead to better generalization, i.e., a smaller difference between the population loss ℒ ⁡ ( θ ) \mathcal{L}(\theta) and the loss on the training set ℒ ^ ​ ( θ ) \widehat{\mathcal{L}}(\theta) . Equivalently, in RL, we would directly expect GR to ensure a similar PR score obtained on the training prompts J ^ ​ ( ϕ , θ ) = 𝔼 s ∼ D tr , a ∼ π ϕ ​ [ R ~ θ ​ ( s , a ) ] \widehat{J}(\phi,\theta)=\mathbb{E}_{s\sim D_{\mathrm{tr}},a\sim\pi_{\phi}}[\widetilde{R}_{\theta}(s,a)] and on the test prompts J ⁡ ( ϕ , θ ) = 𝔼 s ∼ D te , a ∼ π ϕ ​ [ R ~ θ ​ ( s , a ) ] J(\phi,\theta)=\mathbb{E}_{s\sim D_{\mathrm{te}},a\sim\pi_{\phi}}[\widetilde{R}_{\theta}(s,a)] . Instead of considering generalization, we consider the PR accuracy, or more precisely the BT loss ℒ BT \mathcal{L}_{\mathrm{BT}} , necessitating the next two steps.

#### Flat Maxima in Parameter Space imply Robust Maxima in Action Space

We need to connect the flatness of an optimum in parameter space with its flatness in action space, often referred to as robustness. Lee and Yoon (2025) previously investigated this connection under the assumption of a ball with constant reward, which we use as a starting point. While they assume a constant return on a ball B ⁡ ( ϕ ∗ , ℰ ) ≔ { ϵ : ‖ ϵ ‖ ≤ ℰ } B(\phi^{*},\mathcal{E})\coloneq\{\epsilon:\|\epsilon\|\leq\mathcal{E}\} around the optimum ϕ ∗ \phi^{*} , we allow the reward to decrease by at most L ^ \hat{L} :

###### Definition 3.1 ( ℰ − L ^ \mathcal{E}-\widehat{L} flat reward maximum) .

For a reward function R ⁡ ( s , a ) R(s,a) and policy π ϕ ​ ( a | s ) \pi_{\phi}(a|s) , parameterized by ϕ \phi , a maximum ϕ ∗ \phi^{*} is ℰ − L ^ \mathcal{E}-\widehat{L} -flat if the following holds: For all ​ ϵ ∈ ℝ m ​ s.t. ​ ‖ ϵ ‖ ≤ ℰ , \displaystyle\text{For all }\epsilon\in\mathbb{R}^{m}\text{ s.t. }\|\epsilon\|\leq\mathcal{E}, (8) 𝔼 a ∼ π ϕ ∗ ​ ( s ) ​ [ R ~ ​ ( s , a ) ] − 𝔼 a ∼ π ϕ ∗ + ϵ ​ ( s ) ​ [ R ~ ​ ( s , a ) ] ≤ L ^ \displaystyle\mathbb{E}_{a\sim\pi_{\phi^{*}}(s)}\left[\widetilde{R}(s,a)\right]-\mathbb{E}_{a\sim\pi_{\phi^{*}+\epsilon}(s)}\left[\widetilde{R}(s,a)\right]\leq\widehat{L}

We also define the concept of a ( δ , K , ρ ) (\delta,K,\rho) -pairwise robust policy, which measures the probability of action pairs ( a 1 , a 2 ) (a_{1},a_{2}) sampled from the policy violating K / δ K/\delta -Lipschitz-continuity, i.e., | R ~ ​ ( s , a 1 ) − R ~ ​ ( s , a 2 ) | > K , ‖ a 1 − a 2 ‖ < δ |\widetilde{R}(s,a_{1})-\widetilde{R}(s,a_{2})|>K,\|a_{1}-a_{2}\|<\delta :

###### Definition 3.2 ( ( δ , K , ρ ) (\delta,K,\rho) -pairwise robust policy) .

For a policy π ϕ ​ ( a | s ) \pi_{\phi}(a|s) and a reward R ⁡ ( s , a ) R(s,a) , define the sharpness set S K , δ ( R ) := { ( s , a 1 , a 2 ) : ∥ a 1 − a 2 ∥ ≤ δ , | R ( s , a 1 ) − R ( s , a 2 ) | > K } S_{K,\delta}(R):=\{(s,a_{1},a_{2}):\|a_{1}-a_{2}\|\leq\delta,|R(s,a_{1})-R(s,a_{2})|>K\} , i.e., the set of δ \delta -close action pairs for which the reward changes by more than K K . A policy π ϕ \pi_{\phi} is ( δ , K , ρ ) (\delta,K,\rho) -pairwise robust for a reward R R if P ⁡ ( S K , δ ​ ( R ) | π ϕ ) ≤ ρ , P(S_{K,\delta}(R)|\pi_{\phi})\leq\rho\,, (9) with P ⁡ ( S K , δ ​ ( R ) | π ϕ ) P(S_{K,\delta}(R)|\pi_{\phi}) being the probability under s ∼ P ⁡ ( s ) s\sim P(s) and ( a 1 , a 2 ) ∼ i . i . d . π ϕ ​ ( a | s ) (a_{1},a_{2})\stackrel{{\scriptstyle\mathrm{i.i.d.}}}{{\sim}}\pi_{\phi}(a|s) .

We obtain the following proposition, linking flatness in parameter space to ( δ , K , ρ ) (\delta,K,\rho) in action space, under the assumption of a β \beta -smooth PR:

###### Proposition 3.3 ( ℰ − L ^ \mathcal{E}-\widehat{L} flat reward implies ( δ , K , ρ ) (\delta,K,\rho) robust policy) .

Assume a Gaussian policy with fixed covariance Σ \Sigma , where we denote the policy noise Z ∼ 𝒩 ⁡ ( 0 , Σ ) Z\sim\mathcal{N}(0,\Sigma) , and that R ~ ​ ( s , a ) \widetilde{R}(s,a) is β \beta -smooth in a a . Further, 𝔍 ⁡ ( ϕ ∗ ) := ∇ ϕ μ ϕ ​ ( s ) | ϕ = ϕ ∗ \mathfrak{J}(\phi^{*}):=\nabla_{\phi}\,\mu_{\phi}(s)\bigm|_{\phi=\phi^{*}} is the Jacobian matrix of the mean action μ ϕ ​ ( s ) \mu_{\phi}(s) . If ϕ ∗ \phi^{*} is an ℰ − L ^ \mathcal{E}-\widehat{L} flat reward maximum, then the PR action gradient ‖ ∇ a R ~ ​ ( s , μ ϕ ​ ( s ) ) ‖ \|\nabla_{a}\widetilde{R}(s,\mu_{\phi}(s))\| is bounded by G := L ^ D ∗ + β 2 ​ D ∗ + β ​ 𝔼 ​ [ ‖ Z ‖ ] , G:=\frac{\widehat{L}}{D^{*}}+\frac{\beta}{2}D^{*}+\beta\,\mathbb{E}[\|Z\|]\,, with radius D ∗ ≤ ‖ 𝔍 ⁡ ( ϕ ∗ ) ‖ ​ ℰ + 𝒪 ⁡ ( ℰ 2 ) D^{*}\leq\|\mathfrak{J}(\phi^{*})\|\,\mathcal{E}+\mathcal{O}(\mathcal{E}^{2}) . For a given δ > 0 \delta>0 and K > 0 K>0 , with K / δ > G K/\delta>G , we then know that no pairwise violations can occur within the radius r := 1 β ​ ( K δ − G ) r:=\frac{1}{\beta}\bigl(\frac{K}{\delta}-G\bigr) , and thus the policy ϕ ∗ \phi^{*} is ( δ , K , ρ ) (\delta,K,\rho) -robust with P ⁡ ( S K , δ ​ ( R ~ ) ∣ π ϕ ∗ ) ≤ 2 ​ P ​ ( ‖ Z ‖ > r ) ≔ ρ . P(S_{K,\delta}(\widetilde{R})\mid\pi_{\phi^{*}})\leq 2\,P(\|Z\|>r)\coloneq\rho\,.

The proposition follows first linking flatness in parameter space to flatness in action space ( Lee and Yoon, 2025 ) . Then, under a β \beta -smooth PR, by a gradient bound we can ensure the absence of non-Lipschitz action pairs within a radius r r around the mean. Thus a violation requires at least one of the actions to fall outside this region Pr ⁡ ( ‖ Z ‖ > r ) \Pr(\|Z\|>r) and the final result follows from a union bound. A full derivation is shown in Appendix B . Intuitively, for a given maximum, as the sensitivity to disturbances L ^ \hat{L} increases, the probability of overly sharp actions P ⁡ ( S K , δ | π ϕ ) P(S_{K,\delta}|\pi_{\phi}) increases as well. As the radius of the robustness ℰ \mathcal{E} increases, P ⁡ ( S K , δ | π ϕ ) P(S_{K,\delta}|\pi_{\phi}) decreases or stays constant, as we can freely pick a smaller E ′ < E E^{\prime}<E . Thus, flatter, wider minima decrease the risk of sharp action pairs. Next we will show that a larger P ⁡ ( S K , δ | π ϕ ) P(S_{K,\delta}|\pi_{\phi}) incurs a larger excess BT loss ℒ BT \mathcal{L}_{\mathrm{BT}} .

#### Non-Robust Policies Imply Inaccurate Proxy Reward

To connect ( δ , K , ρ ) (\delta,K,\rho) -robustness and the BT loss, we make the assumption of an L L -Lipschitz true reward, | R ∗ ​ ( s , a 1 ) − R ∗ ​ ( s , a 2 ) | ≤ L ​ ‖ a 1 − a 2 ‖ |R^{*}(s,a_{1})-R^{*}(s,a_{2})|\leq L\|a_{1}-a_{2}\| . We then obtain

###### Proposition 3.4 .

For prompts s ∼ P ⁡ ( s ) s\sim P(s) , pairs of actions ( a 1 , a 2 ) (a_{1},a_{2}) , L L -Lipschitz true reward function R ∗ R^{*} , proxy reward R ~ \widetilde{R} , policy π ϕ \pi_{\phi} , and K > L ​ δ K>L\delta , the excess BT loss can be lower bounded as ℒ BT ​ ( R ~ ) − ℒ BT ​ ( R ∗ ) ≥ 2 ​ ( σ ⁡ ( K ) − σ ⁡ ( L ​ δ ) ) 2 ​ P ​ ( S K , δ | π ϕ ) . \mathcal{L}_{\mathrm{BT}}(\widetilde{R})-\mathcal{L}_{\mathrm{BT}}(R^{*})\geq 2(\sigma(K)-\sigma(L\delta))^{2}P(S_{K,\delta}|\pi_{\phi})\,. (10)

The proof is shown in Appendix B.2 . A non-robust policy in action space at least incurs an excess BT loss proportionate to the probability of overly sharp pairs P ⁡ ( S K , δ ) P(S_{K,\delta}) and the magnitude of the sharpness ( σ ⁡ ( K ) − σ ⁡ ( L ​ δ ) ) 2 (\sigma(K)-\sigma(L\delta))^{2} . By changing the policy ϕ \phi to decrease the ratio of violating pairs P ⁡ ( S K , δ ) P(S_{K,\delta}) or the magnitude of the violations K K , we obtain a policy that induces a smaller excess BT loss. As shown above, we can bias the policy updates towards such policies with GR.

#### Limitations

While GR itself can be applied to LMs regardless of whether the action space 𝒜 \mathcal{A} is discrete or continuous, our theoretical argument assumes a Gaussian policy and requires a distance function, which is difficult to define for LMs. As we assume the true reward to be Lipschitz under this distance, a distance under a representation that captures semantic closeness ϕ ⁡ ( a ) : 𝒜 → ℝ d \phi(a):\mathcal{A}\to\mathbb{R}^{d} , such as the hidden space of the LM we are training, would be an appealing option. For example in the illustrative Figure Figure 1 (left), the distance in action corresponds to semantic similarity, not formatting. Further, we only address excess BT loss incurred by overly sharp maxima, i.e., overly sharp PR maxima. We do not show whether or not GR prevents convergence to flat but incorrect regions of the PR.

To empirically validate our theory, we next show that implicit GR, via Reference Resets, can improve RLHF. In Section 5 we leverage explicit GR for further improvements.

## 4 Reference Reset as Gradient Regularization

Instead of performing explicit GR, we can rely on the implicit GR inherent to stochastic gradient descent ( Barrett and Dherin, 2021 ) . We propose to leverage Reference Resets ( Liu et al., 2025a ) where the KL penalty is changed by iteratively resetting its reference to the current policy π ϕ \pi_{\phi} during training i.e., the update is penalized with D KL ​ ( π ϕ , π ϕ ′ ) D_{\mathrm{KL}}(\pi_{\phi};\pi_{\phi^{\prime}}) , where every R R steps we set ϕ ′ ← ϕ \phi^{\prime}\leftarrow\phi . Since implicit GR usually occurs during the later stages of training, we find Reference Resets with a sufficient number of gradient steps per iteration to be an effective way of obtaining flat maxima. While Liu et al. (2025a) reset the policy when the reward has stagnated, we instead choose to perform resets every R R gradient steps similar to prior work using resets in RL ( Nikishin et al., 2022 ; Noukhovitch et al., 2023 ) . We find that it is important to train beyond reward stagnation, as the gradient norm decreases significantly only after stagnation.

#### Setup

We experiment on the well-known TL;DR summarization task ( Stiennon et al., 2020 ) of Reddit posts with human summaries. Following ( Gao et al., 2023 ; Tang et al., 2024 ) we make this a controlled synthetic setup, where the preference and evaluation data is relabeled by a “gold” reward model, Skywork-RewardLlama-3.1-8B-v0.2 ( Liu et al., 2024 ) . Compared to noisy human preferences, this setup gives us an oracle that enables consistent evaluations. We run experiments with models from both Pythia ( Biderman et al., 2023 ) and Qwen 2.5 ( Qwen et al., 2025 ) families for different scales. See full details in Appendix C.2 .

#### Gradient Norm Tracks Sharpness and RM Accuracy

To test our theory of GR, we plot the gradient norm against three important empirical values: 1) The training reward from the RM. 2) The sharpness of the current policy parameters ϕ \phi , which we predict is tied to reward-hacking. The sharpness is estimated by sampling 32 perturbations { ϵ i } i = 1 32 \{\epsilon_{i}\}_{i=1}^{32} and evaluating S ⁡ ( ϕ , θ ) = max i ⁡ J ⁡ ( ϕ , θ ) − J ⁡ ( ϕ + ϵ i , θ ) S(\phi,\theta)=\max_{i}J(\phi,\theta)-J(\phi+\epsilon_{i},\theta) , for each checkpoint. And 3) the BT loss ℒ BT ​ ( θ , ϕ ) \mathcal{L}_{\mathrm{BT}}(\theta,\phi) , which represents how accurate our reward model is for our current policy. We sample completions from our model, label with the gold RM, and get the BT loss under our training RM. In this way, we can empirically check whether we are training in a regime where our PR is accurate. We train a Pythia 1B model with GRPO+Reference Resets and show results in Figure 3 with dashed vertical lines representing resets. In each iteration the gradient norm initially spikes and the reward increases quickly. After the reward stabilizes, the gradient norm decreases. With it the sharpness of the parameters and the BT loss also decrease. This demonstrates that the gradient norm is tied to both the sharpness and accuracy of the PR. Continuing training after the reset, we now start out with a more accurate RM. This enables training in a regime with a good PR, leading to a better final policy.

#### Reference Resets Outperform Standard KL

We run an extensive comparison across both families and two sizes per model on standard baselines. The initial models are SFT-trained on TLDR data. From there we compare our method to DPO ( Rafailov et al., 2023 ) , DPO with reference reset, also known as Trust Region DPO (TR-DPO) ( Gorbatovski et al., 2025 ) , and standard GRPO with a fixed KL penalty. Our results are shown in Table 1 . GRPO + Reference Resets strongly outperforms all other methods for all but one setup. Notably, a Pythia 1B model trained with Reference Resets performs better than a Pythia 2.8B model trained with standard GRPO, even with a finely-tuned β \beta , as shown in Appendix C . In our experiments, TR-DPO’s Reference Resets do not seem to afford the same performance improvements. We speculate that this is because DPO does not use an RM, thus there can be no sharp action-space PR maxima which GR would help to avoid.

#### Is a Scheduled or Weaker β \beta a Sufficient Alternative?

Originally, Reference Resets were proposed ( Liu et al., 2025a ) not to improve RM accuracy but to prevent the KL term from dominating the reward sum. If this was their main mechanism, decreasing the strength of the KL penalty β \beta should have a similar effect. Further, an analysis of the optimal policy under iterated KL constrained optimization (see Appendix E ) shows that the optimal policy for i i reset iterations is equivalent to the optimal policy with a KL constraint to π 1 \pi^{1} with a lower penalty strength β ′ = β / i \beta^{\prime}=\beta/i . However, Figure 4 shows that decreasing β \beta is not able to match Reference Resets. This demonstrates the necessity for our novel insight that Reference Resets change the optimization dynamics via implicit gradient regularization during training and its effects on the PR accuracy.

## 5 Explicit Gradient Regularization