Gradient Regularization Mitigates Reward Hacking in Reinforcement Learning from Human Feedback and Verifiable Rewards

arxiv:2602.18037

Original source · Paper JSON

Post-training · 2026-02-20

Cites (7) · Cited by (0)

Complete cleaned Markdown · Source Markdown including bibliography

Read in parts

1. Part 1
2. Part 2
3. Part 3
4. Part 4
5. Part 5
6. Part 6

Contents

• Gradient Regularization Mitigates Reward Hacking in Reinforcement Learning from Human Feedback and Verifiable Rewards
• Abstract
• Keywords:
• 1 Introduction
• 2 Background
• 2.1 Reinforcement Learning
• 2.2 Proxy Rewards
• Reward Models
• Rule-Based Rewards
• LLM-as-a-Judge
• 2.3 Gradient Regularization (GR)
• 3 Accurate Proxy Rewards via Gradient Regularization
• 3.1 Problem Formulation and Overview
• 3.2 Gradient Regularization Improves Proxy Reward Accuracy
• GR Favors Flat Maxima in Parameter Space
• Flat Maxima in Parameter Space imply Robust Maxima in Action Space
• Definition 3.1 ( ℰ − L ^ \mathcal{E}-\widehat{L} flat reward maximum) .
• Definition 3.2 ( ( δ , K , ρ ) (\delta,K,\rho) -pairwise robust policy) .
• Proposition 3.3 ( ℰ − L ^ \mathcal{E}-\widehat{L} flat reward implies ( δ , K , ρ ) (\delta,K,\rho) robust policy) .
• Non-Robust Policies Imply Inaccurate Proxy Reward
• Proposition 3.4 .
• Limitations
• 4 Reference Reset as Gradient Regularization
• Setup
• Gradient Norm Tracks Sharpness and RM Accuracy
• Reference Resets Outperform Standard KL
• Is a Scheduled or Weaker β \beta a Sufficient Alternative?
• 5 Explicit Gradient Regularization
• 5.1 GR Mitigates Hacking Reward Models in RLHF
• Explicit GR is robust to RM accuracy
• 5.2 GR Prevents Focus on Easy Rule-Based Rewards
• 5.3 GR Mitigates Hacking LLM-as-a-Judge in RLVR
• Limitations
• 6 Conclusion
• Acknowledgements
• Impact Statement
• Appendix A Extended related work
• Reference Resets
• Replacing the KL penalty in Post-training
• Gradient Regularization
• Appendix B Proofs
• B.1 Flatness in Parameter Space implies ( δ , K , ρ ) (\delta,K,\rho) -pairwise robustness
• Definition B.1 ( D − L ^ D-\widehat{L} action robust policy, slightly modified from Lee and Yoon (2025) ) .
• Assumption B.2 (Gaussian policy with fixed covariance and full row-rank Jacobian) .
• Assumption B.3 (Action-smooth proxy reward) .
• Assumption B.4 (Lipschitz-continuous true reward) .
• Proposition B.5 ( ℰ − L ^ \mathcal{E}-\widehat{L} flat return implies D − L ^ D-\widehat{L} robust policy, slightly modified from Lee and Yoon (2025) ) .
• Proof.
• Lemma B.6 (Flatness and β \beta -smoothness imply bounded gradient) .
• Proof.
• Lemma B.7 (Pointwise action-gradient is controlled by Gaussian-smoothing) .
• Proof.
• Lemma B.8 (Pairwise robustness is bounded by local smoothness) .
• Proof.
• Proposition B.9 (From D D - L ^ \widehat{L} robustness to pairwise sharpness control) .
• Proof.
• B.2 BT-Loss Lower Bound Based on Sharpness
• Proposition B.10 .
• Lower Bound
• B.3 Connection of Gradient Regularization and Lipschitz Continuity
• Appendix C Experiment Details
• C.1 Gradient regularization implementation
• C.2 RLHF Details
• C.3 Gradient Regularization on Alpaca Farm
• C.4 Reasoning Experiments
• C.5 LLM Judge Setup
• Appendix D Additional Experiments
• D.1 Gradient Regularization γ \gamma , ε \varepsilon , and learning rate sweeps
• D.2 Gradient regularization decreases BT loss
• D.3 LLM judge ablation
• D.4 KL penalty schedule
• D.5 Steps per reset R R
• D.6 Ablations
• D.7 Runtime Comparison
• D.8 Reward hacking example
• Appendix E Reference Resets in RLHF
• E.1 Asymptotic analysis
• Instructions for reporting errors