Gradient Regularization Mitigates Reward Hacking in Reinforcement Learning from Human Feedback and Verifiable Rewards — part 5/6
Cleaned-text code points [61391, 76810). Bibliography omitted here; source.md includes it.
Let s ∼ P ( s ) s\sim P(s) denote prompts, a ∈ 𝒜 a\in\mathcal{A} denote actions. Let R ∗ ( s , a ) R^{*}(s,a) be the true reward. Under the BT model, the preference probability is p := Pr ( Y = 1 ∣ s , a 1 , a 2 ) = σ ( Δ ∗ ) p:=\Pr(Y=1\mid s,a_{1},a_{2})=\sigma(\Delta_{*}) , Δ ∗ := R ∗ ( s , a 1 ) − R ∗ ( s , a 2 ) \Delta_{*}:=R^{*}(s,a_{1})-R^{*}(s,a_{2}) , with the logistic function σ ( x ) = 1 1 + e − x \sigma(x)=\frac{1}{1+e^{-x}} , where Y = 1 Y=1 means that a 1 a_{1} is preferred over a 2 a_{2} . A parametric reward model R ~ ( s , a ) \widetilde{R}(s,a) induces the predicted pairwise probability q := P R ~ ( Y = 1 | s , a 1 , a 2 ) = σ ( Δ θ ) q:=P_{\widetilde{R}}(Y=1|s,a_{1},a_{2})=\sigma(\Delta_{\theta}) , Δ θ := R ~ ( s , a 1 ) − R ~ ( s , a 2 ) \Delta_{\theta}:=\widetilde{R}(s,a_{1})-\widetilde{R}(s,a_{2}) . The BT loss in equation 3 can be rewritten as ℒ BT π 1 ( θ ) = 𝔼 ( s , a w , a l ) [ − log σ ( R ~ ( s , a w ) − R ~ ( s , a l ) ) ] , \mathcal{L}_{\mathrm{BT}}^{\pi^{1}}(\theta)=\mathbb{E}_{(s,a_{w},a_{l})}\left[-\log\sigma\left(\widetilde{R}(s,a_{\mathrm{w}})-\widetilde{R}(s,a_{\mathrm{l}}\right))\right]\,, (18) but it can also be rewritten to explicitly consider a preference label Y Y , where we have actions a 1 , a 2 a_{1},a_{2} ; then draw Y ∼ Bernoulli ( p ) Y\sim\mathrm{Bernoulli}(p) with p = Pr ( Y = 1 | s , a 1 , a 2 ) p=\Pr(Y=1|s,a_{1},a_{2}) . Taking expectations first over Y | s , a 1 , a 2 Y|s,a_{1},a_{2} and then over ( s , a 1 , a 2 ) (s,a_{1},a_{2}) , we get, ℒ BT π 1 ( θ ) = 𝔼 ( s , a 1 , a 2 ) 𝔼 Y ∼ Bernoulli ( p ) [ ℓ ( q , Y ) ] , \mathcal{L}_{\mathrm{BT}}^{\pi^{1}}(\theta)=\mathbb{E}_{(s,a_{1},a_{2})}\mathbb{E}_{Y\sim\mathrm{Bernoulli}(p)}\left[\ell(q;Y)\right]\,, (19) for Y ∈ { 0 , 1 } Y\in\{0,1\} and ℓ ( q , Y ) = − [ Y log q + ( 1 − Y ) log ( 1 − q ) ] \ell(q;Y)=-\left[Y\log q+(1-Y)\log(1-q)\right] . We proceed with this version.
#### Lower Bound
Condition on ( s , a 1 , a 2 ) (s,a_{1},a_{2}) so that p p is fixed. Then 𝔼 Y ∼ Bernoulli ( p ) [ ℓ ( q , Y ) ] = H ( p , q ) , 𝔼 Y ∼ Bernoulli ( p ) [ ℓ ( p , Y ) ] = H ( p ) , \mathbb{E}_{Y\sim\mathrm{Bernoulli}(p)}\left[\ell(q;Y)\right]=H(p,q),\quad\mathbb{E}_{Y\sim\mathrm{Bernoulli}(p)}\left[\ell(p;Y)\right]=H(p), (20) where H ( p , q ) H(p,q) is the cross-entropy and H ( p ) H(p) is the entropy. Averaging over ( s , a 1 , a 2 ) (s,a_{1},a_{2}) and subtracting yields ℒ ( R ~ ) − ℒ ( R ∗ ) = 𝔼 ( s , a 1 , a 2 ) [ D KL ( Bernoulli ( p ) , Bernoulli ( q ) ) ] . \mathcal{L}(\widetilde{R})-\mathcal{L}(R^{*})=\mathbb{E}_{(s,a_{1},a_{2})}\left[D_{\mathrm{KL}}(\mathrm{Bernoulli}(p);\mathrm{Bernoulli}(q))\right]\,. (21) For a fixed locality parameter δ > 0 \delta>0 and margin threshold K > 0 K>0 , we define the sharpness set S K , δ := { ( s , a 1 , a 2 ) : ∥ a 1 − a 2 ∥ ≤ δ , | Δ θ | > K } . S_{K,\delta}:=\left\{(s,a_{1},a_{2}):\|a_{1}-a_{2}\|\leq\delta,|\Delta_{\theta}|>K\right\}\,. (22) Assume the true reward is L L -Lipschitz: | R ∗ ( s , a 1 ) − R ∗ ( s , a 2 ) | ≤ L ‖ a 1 − a 2 ‖ ⇒ | Δ ∗ | ≤ L δ whenever ‖ a 1 − a 2 ‖ ≤ δ . |R^{*}(s,a_{1})-R^{*}(s,a_{2})|\leq L\|a_{1}-a_{2}\|\Rightarrow|\Delta_{*}|\leq L\delta\quad\text{whenever}\quad\|a_{1}-a_{2}\|\leq\delta\,. (23)
Furthermore, we study the interval separation on S K , δ S_{K,\delta} : On S K , δ S_{K,\delta} we have Δ ∗ ≤ L δ \Delta_{*}\leq L\delta and Δ θ ≥ K \Delta_{\theta}\geq K . By monotonicity and symmetry of σ \sigma , p = σ ( Δ ∗ ) ∈ I ∗ := [ σ ( − L δ ) , σ ( L δ ) ] = [ 1 − σ ( L δ ) , σ ( L δ ) ] , \displaystyle p=\sigma(\Delta_{*})\in I_{*}:=[\sigma(-L\delta),\sigma(L\delta)]=[1-\sigma(L\delta),\sigma(L\delta)], (24) q = σ ( Δ θ ) ∈ I θ := [ 0 , 1 − σ ( K ) ] ∪ [ σ ( K ) , 1 ] , . \displaystyle q=\sigma(\Delta_{\theta})\in I_{\theta}:=[0,1-\sigma(K)]\cup[\sigma(K),1],\,.
If we assume K > L δ K>L\delta , the minimum distance between these sets becomes: inf p ∈ I ∗ , q ∈ I θ | p − q | = min { σ ( K ) − σ ( L δ ) , ( 1 − σ ( L δ ) ) − ( 1 − σ ( K ) ) } = σ ( K ) − σ ( L δ ) . \inf_{p\in I_{*},q\in I_{\theta}}|p-q|=\min\left\{\sigma(K)-\sigma(L\delta),(1-\sigma(L\delta))-(1-\sigma(K))\right\}=\sigma(K)-\sigma(L\delta)\,. (25)
Consequently, for every ( s , a 1 , a 2 ) ∈ S K , δ (s,a_{1},a_{2})\in S_{K,\delta} , | p − q | ≥ σ ( K ) − σ ( L δ ) . |p-q|\geq\sigma(K)-\sigma(L\delta)\,. (26) Since we can use the inequality D KL ( Bernoulli ( p ) , Bernoulli ( q ) ) ≥ 2 | p − q | 2 D_{\mathrm{KL}}(\mathrm{Bernoulli}(p);\mathrm{Bernoulli}(q))\geq 2|p-q|^{2} for p , q ∈ ( 0 , 1 ) p,q\in(0,1) , combining with equation 26 yields: D KL ( Bernoulli ( p ) , Bernoulli ( q ) ) ≥ 2 ( σ ( K ) − σ ( L δ ) ) 2 for every ( s , a 1 , a 2 ) ∈ S K , δ . D_{\mathrm{KL}}(\mathrm{Bernoulli}(p);\mathrm{Bernoulli}(q))\geq 2(\sigma(K)-\sigma(L\delta))^{2}\quad\text{for every}\quad(s,a_{1},a_{2})\in S_{K,\delta}\,. (27)
Let 𝟏 S := 𝟏 { Z ∈ S K , δ } \mathbf{1}_{S}:=\mathbf{1}\{Z\in S_{K,\delta}\} indicate membership of S K , δ S_{K,\delta} . We can make equation 27 valid for all ( s , a 1 , a 2 ) (s,a_{1},a_{2}) by multiplying the RHS with the indicator:
D KL ( Bernoulli ( p ) , Bernoulli ( q ) ) ≥ 2 ( σ ( K ) − σ ( L δ ) ) 2 𝟏 S for all ( s , a 1 , a 2 ) . D_{\mathrm{KL}}(\mathrm{Bernoulli}(p);\mathrm{Bernoulli}(q))\geq 2(\sigma(K)-\sigma(L\delta))^{2}\mathbf{1}_{S}\quad\text{for all}\quad(s,a_{1},a_{2})\,. (28) By taking the expectation of equation 28 and using equation 21 , we obtain the desired lower bound ℒ BT ( R ~ ) − ℒ BT ( R ∗ ) ≥ 2 ( σ ( K ) − σ ( L δ ) ) 2 P ( S K , δ ) . \mathcal{L}_{\mathrm{BT}}(\widetilde{R})-\mathcal{L}_{\mathrm{BT}}(R^{*})\geq 2(\sigma(K)-\sigma(L\delta))^{2}P(S_{K,\delta})\,. (29)
### B.3 Connection of Gradient Regularization and Lipschitz Continuity
For completeness, we reproduce the argument of Zhao et al. (2022) , which explicitly connects gradient regularization to Lipschitzness in parameters θ \theta .
By the mean value theorem for differentiable L L , we have L ( θ 1 ) − L ( θ 2 ) = ∇ L ( ζ ) T ( θ 1 − θ 2 ) , L(\theta_{1})-L(\theta_{2})=\nabla L(\zeta)^{T}(\theta_{1}-\theta_{2})\,, with ζ = c θ 1 + ( 1 − c ) θ 2 \zeta=c\theta_{1}+(1-c)\theta_{2} , with some c ∈ [ 0 , 1 ] c\in[0,1] and the Cauchy-Schwarz inequality then yields ‖ L ( θ 1 ) − L ( θ 2 ) ‖ ≤ ‖ ∇ L ( ζ ) ‖ ‖ ( θ 1 − θ 2 ) ‖ . \|L(\theta_{1})-L(\theta_{2})\|\leq\|\nabla L(\zeta)\|\|(\theta_{1}-\theta_{2})\|\,. Here, ‖ ∇ L ( ζ ) ‖ ||\nabla L(\zeta)|| takes the role of the Lipschitz constant and as θ 2 → θ 1 \theta_{2}\to\theta_{1} , ‖ ∇ L ( ζ ) ‖ ||\nabla L(\zeta)|| becomes ‖ ∇ L ( θ ) ‖ ||\nabla L(\theta)|| . Thus we can see that gradient regularization leads to local Lipschitzness in parameter space, i.e., a flat local minimum.
## Appendix C Experiment Details
In this section we provide additional experimental details.
### C.1 Gradient regularization implementation
In our experiments, we use the GR method ( Karakida et al., 2023 ) based on the finite difference estimate Δ ϕ ‖ ∇ ϕ ℒ ( ϕ ) ‖ 2 = ∇ ϕ ℒ ( ϕ + ε ∇ ϕ ℒ ( ϕ ) ) − ∇ ϕ ℒ ( ϕ ) ε . \Delta_{\phi}\|\nabla_{\phi}\mathcal{L}(\phi)\|^{2}=\frac{\nabla_{\phi}\mathcal{L}(\phi+\varepsilon\nabla_{\phi}\mathcal{L}(\phi))-\nabla_{\phi}\mathcal{L}(\phi)}{\varepsilon}\,. We, thus, need to perturb the parameters ϕ \phi . Empirically, we found it to be beneficial to perturb only the parameters of the transformer blocks, including attention matrices, MLP weights and layer norm parameters, but not perturb the embedding layer or final output layer. In principle, to calculate ‖ ∇ ϕ J ( ϕ + ∇ ϕ J ( ϕ ) ) ‖ \|\nabla_{\phi}J(\phi+\nabla_{\phi}J(\phi))\| we would also need to sample new actions a i ∼ π ϕ + ∇ ϕ J ( ϕ ) ( a | s ) a_{i}\sim\pi_{\phi+\nabla_{\phi}J(\phi)}(a|s) and estimate the gradient using these. In practice, the computational overhead of this would be large, we reuse the same actions. This introduces some bias which could be corrected using importance sampling, however, empirically we found this to be unnecessary. We also need to choose a perturbation strength ε \varepsilon . While Karakida et al. (2023) found a relatively large ε ≈ 0.05 \varepsilon\approx 0.05 to perform best, initial experiments showed ε = 10 − 3 \varepsilon=10^{-3} to perform well in our setting. We thus used it through-out our experiments. We further use gradient clipping, both for the disturbance ∇ ϕ J ( ϕ ) \nabla_{\phi}J(\phi) and the gradient ‖ ∇ ϕ J ( ϕ + ∇ ϕ J ( ϕ ) ) ‖ \|\nabla_{\phi}J(\phi+\nabla_{\phi}J(\phi))\| , each to 10. We found this to prevent gradient spikes from destabilizing training. The final combined gradient is then again clipped to 1.0 within DeepSpeed, as is done for the non GR methods as well. We train our models using DeepSpeed ZeRO 2 ( Rajbhandari et al., 2020 ) and use gradient accumulation.
### C.2 RLHF Details
For our RLHF experiments, we first need to train an SFT model, sample a training dataset D RM D_{\mathrm{RM}} , and train an RM. For both, we use the code and hyper-parameters provided by Huang et al. (2024) , which use the AdamW optimizer ( Loshchilov and Hutter, 2019 ) with weight decay. The SFT models are trained on the SFT dataset for one epoch. For the summarization experiments, we use the SFT dataset provided by ( Huang et al., 2024 ) , for Alpaca experiments we use the Alpaca-Instructions dataset ( Dubois et al., 2023 ) , but filter it by length following ( Ackermann et al., 2025 ) . This length filtering to a maximum length of 512 tokens significantly decreases the computational cost. Further, while Alpaca-Instructions contains separate splits for SFT, RM and RL training, we combine them to a single dataset as used in the summarization setting. To obtain the RM training dataset, we sample pairs of responses from the SFT model and label them with either the Gold RM “Skywork-RewardLlama-3.1-8B-v0.2” ( Liu et al., 2024 ) for summarization or with GPT4.1-Nano for Alpaca experiments. For the summarization task we use 278,496 preference pairs, for Alpaca Experiments we use 43,008 pairs for the 3B model and 86,016 pairs for the 0.5B and 1.5B models. With the smaller data amount the 0.5B and 1.5B models did not produce sufficiently accurate RMs to use for subsequent GRPO updates. We train the RMs, initialized from the SFT model, for one epoch on the dataset with the hyper-parameters as recommended by ( Huang et al., 2024 ) . We then use the Dr.GRPO implementation provided by TRL ( von Werra et al., 2020 ) with the hyper-parameters as listed in Table 5 . Additionally, we tuned the KL penalty strength β \beta and GR strength γ \gamma as listed Tables 6 and 7 . We use the standard KL penalty estimate provided by TRL, including the KL penalty in the loss rather than in the reward.
### C.3 Gradient Regularization on Alpaca Farm
We perform on the AlpacaFarm dataset ( Dubois et al., 2023 ) with Qwen2.5 models ( Qwen et al., 2025 ) . Similar to the TL;DR experiments we first train an SFT model, sample pairs of responses from it, and obtain pairwise comparisons from GPT4.1 Nano. We then train an RM based on these comparisons, initialized from the SFT model.
### C.4 Reasoning Experiments
We use the hyperparameters as stated in Table 8 and Table 9 . We did not tune β \beta , ϵ \epsilon or γ \gamma and simply used conservative values from the RLHF experiments. For both GR and no regularization we tried learning rates 3 × 10 − 6 3\times 10^{-6} and 5 × 10 − 6 5\times 10^{-6} . For GR, the larger learning rate was beneficial.
We follow the setup of Wei et al. (2025) , with GSM8K using one-shot prompting, math uses chain-of-thought prompting, both with the default Qwen2.5-Instruct system instructions. We also use the same reward terms, which are based on the rewards used by Open R1 ( Hugging Face, 2025 ) . For GSM8K, the prompt is Respond in the following format: <reasoning>...</reasoning><answer>...</answer> , followed by a one-shot example. The reward consists of a rule-based correctness reward ( [ 0 , 2 ] [0,2] ), a formatting term checking whether the answer is an integer formatting reward ( { 0 , 0.5 } \{0,0.5\} ), a formatting term checking whether the formatting is exactly followed including whitespace ( { 0 , 1.0 } \{0,1.0\} ), and excluding whitespace ( { 0 , 1.0 } \{0,1.0\} ), and a formatting reward for matching XML tags ( { 0 , 0.5 } \{0,0.5\} ).
On MATH the prompt is simply the problem statement followed by Let's think step by step and output the final answer within \\boxed{}. . The used reward consists of a correctness reward ( [ 0 , 2 ] [0,2] ) and a formatting reward rewarding up to three of the following with 1 / 3 1/3 each: Step # keywords, Numbered lists, bullet points, First, Second, Next, Finally keywords. This is done to encourage reasoning ( Hugging Face, 2025 ) .
### C.5 LLM Judge Setup
In the main text we use Qwen2.5 1.5B-Instruct ( Qwen et al., 2025 ) as a judge with the following prompt:
⬇
We additionally use 1-shot prompting with a correct example. During training the agent is provided the correctness score scaled to [ 0 , 2 ] [0,2] , along with the same formatting rewards used in the rule-based-reward setting.
## Appendix D Additional Experiments
### D.1 Gradient Regularization γ \gamma , ε \varepsilon , and learning rate sweeps
To evaluate the sensitivity of GR to the hyper-parameters controlling strength of the regularization γ \gamma and the strength of the perturbation ε \varepsilon , we performed experiments on the Qwen2.5 1.5B model in the Alpaca GPT4.1 Nano setting. We fix either γ = 3 × 10 − 2 \gamma=3\times 10^{-2} or ε = 10 − 3 \varepsilon=10^{-3} and vary the respective other parameter. We show the results in Figure 11 . We find the performance to be rather predictable, facilitating hyper-parameter tuning. We used the same disturbance strength ε = 10 − 3 \varepsilon=10^{-3} for all experiments and did not find it necessary to tune it per task or model. In the reasoning experiments we did not tune γ \gamma and simply used γ = 10 − 3 \gamma=10^{-3} , however, in the RLHF experiments we found it necessary to tune γ \gamma as described above, just like we found it necessary to tune β \beta for the KL penalty.
We also perform a learning rate sweep for the KL penalty and GR in the same Alpaca, Qwen2.5 1.5B setting. We use the best performing KL hyper-parameters from our main hyper-parameter optimization, i.e., β = 0.1 \beta=0.1 for the KL penalty and γ = 0.03 \gamma=0.03 for GR. The results are shown in Section D.1 .
### D.2 Gradient regularization decreases BT loss
We also perform experiments in the synthetic TL;DR gold model setup, training a Qwen 2.5 0.5B model with different GR strengths γ \gamma . The results in Figure 12 show that stronger regularization leads to a lower BT model loss ℒ BT ( ϕ , θ ) \mathcal{L}_{\mathrm{BT}}(\phi,\theta) , evaluated on 4096 action pairs with labels from the gold reward model. Interestingly, training with strong GR can result in a decreasing BT loss ℒ BT \mathcal{L}_{\mathrm{BT}} beyond the initial BT loss, which we did not observe when utilizing either KL regularization or Reference Resets. This illustrates the practical strength of the connection between gradient norm and PR accuracy.
### D.3 LLM judge ablation