Gradient Regularization Mitigates Reward Hacking in Reinforcement Learning from Human Feedback and Verifiable Rewards — part 4/6

Cleaned-text code points [46164, 61391). Bibliography omitted here; source.md includes it.

###### Definition B.1 ( D − L ^ D-\widehat{L} action robust policy, slightly modified from Lee and Yoon (2025) ) .

For a reward function R ⁡ ( s , a ) R(s,a) and policy π ϕ ​ ( a | s ) \pi_{\phi}(a|s) , parameterized by ϕ \phi , a maximum ϕ ∗ \phi^{*} is D D - L ^ \widehat{L} action robust if the following holds: For all ​ δ ∈ ℝ | A | ​ s.t. ​ ‖ δ ‖ ≤ D : 𝔼 a ∼ π ϕ ∗ ​ [ R ~ ​ ( s , a ) ] − 𝔼 a ∼ π ϕ ∗ ​ [ R ~ ​ ( s , a + δ ) ] ≤ L ^ \displaystyle\text{For all }\delta\in\mathbb{R}^{|A|}\text{ s.t. }\|\delta\|\leq D:\quad\quad\quad\mathbb{E}_{a\sim\pi_{\phi^{*}}}\left[\widetilde{R}(s,a)\right]-\mathbb{E}_{a\sim\pi_{\phi^{*}}}\left[\widetilde{R}(s,a+\delta)\right]\leq\widehat{L} (11)

We will also need the following assumptions during this section:

###### Assumption B.2 (Gaussian policy with fixed covariance and full row-rank Jacobian) .

For each s s , π ϕ ∗ ( ⋅ ∣ s ) = 𝒩 ( μ ϕ ∗ ( s ) , Σ ) \pi_{\phi^{*}}(\cdot\mid s)=\mathcal{N}(\mu_{\phi^{*}}(s),\Sigma) with a fixed positive definite covariance Σ \Sigma . The Jacobian matrix 𝔍 ⁡ ( ϕ ∗ ) \mathfrak{J}(\phi^{*}) of the mean action μ ϕ ​ ( s ) \mu_{\phi}(s) w.r.t. ϕ \phi , evaluated at ϕ ∗ \phi^{*} , has full row rank.

###### Assumption B.3 (Action-smooth proxy reward) .

For all s ∈ 𝒮 s\in\mathcal{S} , the PR R ~ ​ ( s , a ) \widetilde{R}(s,a) is β \beta -smooth in a a , i.e., ‖ ∇ a R ~ ​ ( s , a 1 ) − ∇ a R ~ ​ ( s , a 2 ) ‖ ≤ β ​ ‖ a 1 − a 2 ‖ \|\nabla_{a}\widetilde{R}(s,a_{1})-\nabla_{a}\widetilde{R}(s,a_{2})\|\leq\beta\|a_{1}-a_{2}\| for all a 1 , a 2 ∈ 𝒜 a_{1},a_{2}\in\mathcal{A} .

###### Assumption B.4 (Lipschitz-continuous true reward) .

For all s ∈ 𝒮 s\in\mathcal{S} , the true reward R ∗ ​ ( s , a ) R^{*}(s,a) is Lipschitz in a a , i.e., | R ∗ ​ ( s , a 1 ) − R ∗ ​ ( s , a 2 ) | ≤ L ​ ‖ a 1 − a 2 ‖ |R^{*}(s,a_{1})-R^{*}(s,a_{2})|\leq L\|a_{1}-a_{2}\| for all a 1 , a 2 a_{1},a_{2} .

We now can show that an ℰ − L ^ \mathcal{E}-\widehat{L} flat return implies D − L ^ D-\widehat{L} robust policy:

###### Proposition B.5 ( ℰ − L ^ \mathcal{E}-\widehat{L} flat return implies D − L ^ D-\widehat{L} robust policy, slightly modified from Lee and Yoon (2025) ) .

If ϕ ∗ \phi^{*} is an ℰ − L ^ \mathcal{E}-\widehat{L} flat return maximum of a policy under assumption B.2 , then the policy ϕ ∗ \phi^{*} is D ∗ − L ^ D^{*}-\widehat{L} robust, where: D ∗ ≤ | 𝔍 ⁡ ( ϕ ∗ ) | ℰ + 𝒪 ⁡ ( ℰ 2 ) , D^{*}\leq\bigl\lVert\mathfrak{J}(\phi^{*})\bigr\rVert\,\mathcal{E}+\mathcal{O}(\mathcal{E}^{2}), (12) and 𝔍 ⁡ ( ϕ ∗ ) := ∇ ϕ μ ϕ ​ ( s ) | ϕ = ϕ ∗ \mathfrak{J}(\phi^{*}):=\nabla_{\phi}\,\mu_{\phi}(s)\bigm|_{\phi=\phi^{*}} is the Jacobian matrix of the mean action μ ϕ ​ ( s ) \mu_{\phi}(s) with respect to ϕ \phi , evaluated at ϕ ∗ \phi^{*} .

###### Proof.

Assume a Gaussian policy with fixed covariance Σ \Sigma , such that π ϕ ​ ( a | s ) = 𝒩 ⁡ ( a , μ ϕ ​ ( s ) , Σ ) \pi_{\phi}(a|s)=\mathcal{N}(a;\mu_{\phi}(s),\Sigma) , where μ ϕ ​ ( s ) \mu_{\phi}(s) is the mean of the Gaussian we are training. A Taylor expansion yields μ ϕ + ϵ ​ ( s ) = μ ϕ ​ ( s ) + J ⁡ ( ϕ ) ​ ϵ + O ⁡ ( ‖ ϵ ‖ 2 ) \mu_{\phi+\epsilon}(s)=\mu_{\phi}(s)+J(\phi)\epsilon+O(||\epsilon||^{2}) .

Define the change in policy for a given state s s due to the perturbation ϵ \epsilon as δ ⁡ ( s ) = μ ϕ + ϵ ​ ( s ) − μ ϕ ​ ( s ) = J ⁡ ( ϕ ) ​ ϵ + O ⁡ ( ‖ ϵ ‖ 2 ) . \delta(s)=\mu_{\phi+\epsilon}(s)-\mu_{\phi}(s)=J(\phi)\epsilon+O(||\epsilon||^{2})\,. Then by the triangle inequality, the Cauchy-Schwarz inequality and definition ‖ ϵ ‖ < ℰ \|\epsilon\|<\mathcal{E} , we have ‖ δ ⁡ ( s ) ‖ ≤ ‖ J ⁡ ( ϕ ) ‖ ​ ‖ ϵ ‖ + O ⁡ ( ‖ ϵ ‖ 2 ) ≤ | | J ⁡ ( ϕ ) | | ℰ + O ⁡ ( ℰ 2 ) . ||\delta(s)||\leq||J(\phi)||||\epsilon||+O(||\epsilon||^{2})\leq||J(\phi)||\mathcal{E}+O(\mathcal{E}^{2})\,.

The sub-optimality of this action perturbation δ \delta is 𝔼 s ∼ P ⁡ ( s ) , a ∼ π ϕ ∗ ​ ( s ) ​ [ R ~ ​ ( s , a ) ] − 𝔼 s ∼ P ⁡ ( s ) , a ∼ π ϕ ∗ ​ ( s ) ​ [ R ~ ​ ( s , a + δ ⁡ ( s ) ) ] \displaystyle\mathbb{E}_{s\sim P(s),a\sim\pi_{\phi^{*}}(s)}\left[\widetilde{R}(s,a)\right]-\mathbb{E}_{s\sim P(s),a\sim\pi_{\phi^{*}}(s)}\left[\widetilde{R}(s,a+\delta(s))\right] (13) = Def. δ \displaystyle\stackrel{{\scriptstyle\text{Def. $\delta$}}}{{=}} 𝔼 s ∼ P ⁡ ( s ) , a ∼ π ϕ ∗ ​ ( s ) ​ [ R ~ ​ ( s , a ) ] − 𝔼 s ∼ P ⁡ ( s ) , a ∼ π ϕ ∗ + ϵ ​ ( s ) ​ [ R ~ ​ ( s , a ) ] ≤ Def 8 L ^ \displaystyle\mathbb{E}_{s\sim P(s),a\sim\pi_{\phi^{*}}(s)}\left[\widetilde{R}(s,a)\right]-\mathbb{E}_{s\sim P(s),a\sim\pi_{\phi^{*}+\epsilon}(s)}\left[\widetilde{R}(s,a)\right]\stackrel{{\scriptstyle\text{Def \ref{def:e-l-flatrew}}}}{{\leq}}\widehat{L} (14)

Thus a ℰ − L ^ \mathcal{E}-\widehat{L} flat reward maximum implies a D ∗ − L ^ D^{*}-\widehat{L} robust policy with D ∗ ≤ ‖ J ⁡ ( ϕ ∗ ) ‖ ​ ℰ + 𝒪 ⁡ ( ℰ 2 ) D^{*}\leq||J(\phi^{*})||\mathcal{E}+\mathcal{O}(\mathcal{E}^{2}) ∎

As we want to show an excess BT loss based on a Lipschitz-continuity assumption, which is defined over action pairs, we need to relate D − L ^ D-\widehat{L} robustness to P ⁡ ( S K , δ | π ϕ ) P(S_{K,\delta}|\pi_{\phi}) . For this purpose we need B.3 following two simple lemmas:

###### Lemma B.6 (Flatness and β \beta -smoothness imply bounded gradient) .

Let f : ℝ d → ℝ f:\mathbb{R}^{d}\to\mathbb{R} be differentiable and β \beta -smooth on a ball B ⁡ ( c , r ) B(c,r) , i.e., ‖ ∇ f ​ ( x ) − ∇ f ​ ( y ) ‖ ≤ β ​ ‖ x − y ‖ ∀ x , y ∈ B ⁡ ( c , r ) \|\nabla f(x)-\nabla f(y)\|\leq\beta\|x-y\|\quad\forall x,y\in B(c,r) . Assume a "flat maximum" in c c , such that f ⁡ ( c ) − f ⁡ ( c + u ) ≤ L ^ ∀ u : ‖ u ‖ ≤ r f(c)-f(c+u)\leq\widehat{L}\quad\forall u:\|u\|\leq r . Then the gradient norm at c is bounded as ‖ ∇ f ​ ( c ) ‖ ≤ L ^ r + β 2 ​ r . \|\nabla f(c)\|\leq\frac{\widehat{L}}{r}+\frac{\beta}{2}r\,.

###### Proof.

Based on the standard quadratic upper bound based on β \beta -smoothness, we know f ( c + u ) ≤ f ( c ) + ∇ f ( c ) T u + β 2 ∥ u ∥ 2 ∀ c + u ∈ B ( c , r ) . f(c+u)\leq f(c)+\nabla f(c)^{T}u+\frac{\beta}{2}\|u\|^{2}\quad\forall c+u\in B(c,r)\,. We can rearrange this to f ( c ) − f ( c + u ) ≥ − ∇ f ( c ) T u − β 2 ∥ u ∥ 2 . f(c)-f(c+u)\geq-\nabla f(c)^{T}u-\frac{\beta}{2}\|u\|^{2}\,. By choosing the worst case u = − r ​ ∇ f ​ ( c ) ‖ ∇ f ​ ( c ) ‖ u=-r\frac{\nabla f(c)}{\|\nabla f(c)\|} (if ‖ ∇ f ​ ( c ) ‖ = 0 \|\nabla f(c)\|=0 the bound is trivially true), we have ‖ u ‖ = r \|u\|=r and − ∇ f ( c ) T u = r ∥ ∇ f ( c ) ∥ -\nabla f(c)^{T}u=r\|\nabla f(c)\| . Plugging this into the inequality and using f ⁡ ( c ) − f ⁡ ( c + u ) ≤ L ^ f(c)-f(c+u)\leq\widehat{L} yields ‖ ∇ f ​ ( c ) ‖ ≤ L ^ r + β 2 ​ r . \|\nabla f(c)\|\leq\frac{\widehat{L}}{r}+\frac{\beta}{2}r\,. ∎

Note that robustness with radius r r implies robustness with any r ′ < r r^{\prime}<r and we could further improve this bound by picking r ′ = min ⁡ ( 2 ​ β ​ L ^ ) ≔ r ∗ r^{\prime}=\min(\sqrt{2\beta\hat{L}})\coloneq r^{*} , if r ∗ ≤ r r^{*}\leq r . We do not do this here for ease of exposition. We next need to connect the gradient bound of the expected 𝔼 ​ [ R ~ ​ ( s , a + Z ) ] \mathbb{E}[\widetilde{R}(s,a+Z)] to a bound on the gradient of the pointwise R ~ ​ ( s , a ) \widetilde{R}(s,a) :

###### Lemma B.7 (Pointwise action-gradient is controlled by Gaussian-smoothing) .

Fix s s and let f ​ ( a ) = R ~ ​ ( s , a ) f(a)=\widetilde{R}(s,a) . Under Assumption B.3 and for Z ∼ 𝒩 ⁡ ( 0 , Σ ) Z\sim\mathcal{N}(0,\Sigma) , define the Gaussian-smoothed reward f ¯ ​ ( c ) := 𝔼 ​ [ f ​ ( c + Z ) ] . \bar{f}(c):=\mathbb{E}[f(c+Z)]\,. Then ∇ f ¯ ​ ( c ) = 𝔼 ⁡ [ ∇ f ​ ( c + Z ) ] \nabla\bar{f}(c)=\mathbb{E}[\nabla f(c+Z)] and for all c c , ‖ ∇ f ​ ( c ) ‖ ≤ ‖ ∇ f ¯ ​ ( c ) ‖ + β ​ 𝔼 ​ [ ‖ Z ‖ ] . \|\nabla f(c)\|\leq\|\nabla\bar{f}(c)\|+\beta\,\mathbb{E}\left[\|Z\|\right]\,.

###### Proof.

Since f f is β \beta -smooth, it is continuously differentiable with Lipschitz gradient, and we can interchange gradient and expectation to obtain ∇ f ¯ ​ ( c ) = 𝔼 ⁡ [ ∇ f ​ ( c + Z ) ] \nabla\bar{f}(c)=\mathbb{E}[\nabla f(c+Z)] . Then, by Jensen’s inequality and β \beta -smoothness, ‖ ∇ f ​ ( c ) − ∇ f ¯ ​ ( c ) ‖ = ‖ 𝔼 ⁡ [ ∇ f ​ ( c ) − ∇ f ​ ( c + Z ) ] ‖ ≤ 𝔼 ⁡ [ ‖ ∇ f ​ ( c ) − ∇ f ​ ( c + Z ) ‖ ] ≤ β ​ 𝔼 ​ ‖ Z ‖ , \|\nabla f(c)-\nabla\bar{f}(c)\|=\Bigl\|\mathbb{E}[\nabla f(c)-\nabla f(c+Z)]\Bigr\|\leq\mathbb{E}\bigl[\|\nabla f(c)-\nabla f(c+Z)\|\bigr]\leq\beta\,\mathbb{E}\|Z\|\,, then, by the triangle inequality and the previous line, ‖ ∇ f ​ ( c ) ‖ \displaystyle\|\nabla f(c)\| = ‖ ∇ f ​ ( c ) − ∇ f ​ ( c + Z ) + ∇ f ​ ( c + Z ) ‖ ≤ ‖ ∇ f ​ ( c + Z ) ‖ + ‖ ∇ f ​ ( c ) − ∇ f ​ ( c + Z ) ‖ \displaystyle=\|\nabla f(c)-\nabla f(c+Z)+\nabla f(c+Z)\|\leq\|\nabla f(c+Z)\|+\|\nabla f(c)-\nabla f(c+Z)\| (15) ≤ ‖ ∇ f ​ ( c + Z ) ‖ + β ​ 𝔼 ​ [ ‖ Z ‖ ] = ‖ ∇ f ¯ ​ ( c ) ‖ + β ​ 𝔼 ​ [ ‖ Z ‖ ] . \displaystyle\leq\|\nabla f(c+Z)\|+\beta\mathbb{E}\left[\|Z\|\right]=\|\nabla\bar{f}(c)\|+\beta\mathbb{E}\left[\|Z\|\right]\,. (16)

∎

Then, we need to connect local smoothness and our knowledge of the gradient norm ‖ ∇ f ​ ( c ) ‖ \|\nabla f(c)\| at the center to violations of Lipschitz-continuity:

###### Lemma B.8 (Pairwise robustness is bounded by local smoothness) .

Fix a state s s . Denote f ​ ( a ) = R ~ ​ ( s , a ) f(a)=\widetilde{R}(s,a) . Assume that there exists a center c ∈ A c\in A and radius r > 0 r>0 such that f f is β \beta -smooth on the ball B ⁡ ( c , r ) := { a : ‖ a − c ‖ ≤ r } B(c,r):=\{a:\|a-c\|\leq r\} . Let a , a 1 , a 2 ∼ i . i . d . π ⁡ ( a | s ) a,a_{1},a_{2}\stackrel{{\scriptstyle\mathrm{i.i.d.}}}{{\sim}}\pi(a|s) .

Then for any δ > 0 \delta>0 and K > 0 K>0 , we have P ⁡ ( ‖ a 1 − a 2 ‖ ≤ δ , | f ⁡ ( a 1 ) − f ⁡ ( a 2 ) | > K ) ≤ 2 ​ P ​ ( ‖ a − c ‖ > r ) , P(\|a_{1}-a_{2}\|\leq\delta,|f(a_{1})-f(a_{2})|>K)\leq 2P(\|a-c\|>r)\,, for r = 1 β ​ ( K δ − ‖ ∇ f ​ ( x ) ‖ ) r=\frac{1}{\beta}\left(\frac{K}{\delta}-\|\nabla f(x)\|\right)

###### Proof.

For any x ∈ B ⁡ ( c , r ) x\in B(c,r) , by β \beta -smoothness, ‖ ∇ f ​ ( x ) ‖ ≤ ‖ ∇ f ​ ( c ) ‖ + ‖ ∇ f ​ ( x ) − ∇ f ​ ( c ) ‖ ≤ ‖ ∇ f ​ ( c ) ‖ + β ​ ‖ x − c ‖ ≤ | ∇ f ​ ( c ) | + β ​ r . \|\nabla f(x)\|\leq\|\nabla f(c)\|+\|\nabla f(x)-\nabla f(c)\|\leq\|\nabla f(c)\|+\beta\|x-c\|\leq\|\nabla f(c)\|+\beta r\,. For any two points x , y ∈ B ⁡ ( c , r ) x,y\in B(c,r) , f ( x ) − f ( y ) = ∫ 0 1 ∇ f ( y + t ( x − y ) ) T ( x − y ) d t . f(x)-f(y)=\int_{0}^{1}\nabla f(y+t(x-y))^{T}(x-y)dt\,. taking the norm, using the Cauchy-Schwarz inequality, and the bounded gradient, we get | f ⁡ ( x ) − f ⁡ ( y ) | ≤ ∫ 0 1 ‖ ∇ f ​ ( y + t ⁡ ( x − y ) ) ‖ | x − y | 𝑑 t ≤ ( ‖ ∇ f ​ ( c ) ‖ + β ​ r ) ​ ‖ x − y ‖ . |f(x)-f(y)|\leq\int_{0}^{1}\|\nabla f(y+t(x-y))\|\|x-y\|dt\,\leq(\|\nabla f(c)\|+\beta r)\|x-y\|\,. Therefore, if x , y ∈ B ⁡ ( c , r ) x,y\in B(c,r) and ‖ x − y ‖ ≤ δ \|x-y\|\leq\delta , then | f ⁡ ( x ) − f ⁡ ( y ) | ≤ ( ‖ ∇ f ​ ( c ) ‖ + β ​ r ) ​ δ . |f(x)-f(y)|\leq(\|\nabla f(c)\|+\beta r)\delta\,. Thus, the event { ‖ a 1 − a 2 ‖ ≤ δ , | f ⁡ ( a 1 ) − f ⁡ ( a 2 ) | > K } \{\|a_{1}-a_{2}\|\leq\delta,|f(a_{1})-f(a_{2})|>K\} cannot occur when a 1 , a 2 ∈ B ⁡ ( c , r ) a_{1},a_{2}\in B(c,r) and ( ‖ ∇ f ​ ( c ) ‖ + β ​ r ) ​ δ < K (\|\nabla f(c)\|+\beta r)\delta<K . It thus can only occur if at least one of a 1 a_{1} or a 2 a_{2} is not in B ⁡ ( c , r ) B(c,r) , which occurs with probability P ⁡ ( ‖ a − c ‖ > r ) P(\|a-c\|>r) . By a union bound of the two events we get the result. ∎

Finally, we can put it all together:

###### Proposition B.9 (From D D - L ^ \widehat{L} robustness to pairwise sharpness control) .

Assume a Gaussian policy with full row-rank Jacobian (Assumption B.2 ), a β \beta -smooth proxy reward R ~ \widetilde{R} ( B.3 ), and fix a state s s . If ϕ ∗ \phi^{*} is D D - L ^ \widehat{L} action robust, then for any δ > 0 \delta>0 and K > 0 K>0 such that K / δ > G K/\delta>G , we have gradient magnitude bound G G and non-violating radius r r , G := L ^ D + β 2 ​ D + β ​ 𝔼 ​ ‖ Z ‖ , r := 1 β ​ ( K δ − G ) , G:=\frac{\widehat{L}}{D}+\frac{\beta}{2}D+\beta\,\mathbb{E}\|Z\|\,,\quad r:=\frac{1}{\beta}\bigl(\frac{K}{\delta}-G\bigr)\,,

such that P ⁡ ( S K , δ ​ ( R ~ ) ∣ π ϕ ∗ ) ≤ 2 ​ P ​ ( ‖ Z ‖ > r ) . P(S_{K,\delta}(\widetilde{R})\mid\pi_{\phi^{*}})\leq 2\,P(\|Z\|>r)\,.

###### Proof.

Fix s s and denote c = μ ϕ ∗ ​ ( s ) c=\mu_{\phi^{*}}(s) . Denote the PR as f ​ ( a ) = R ~ ​ ( s , a ) f(a)=\widetilde{R}(s,a) , mean action as c = μ ϕ ∗ ​ ( s ) c=\mu_{\phi^{*}}(s) , and the smoothed proxy reward f ¯ ​ ( c ) = 𝔼 Z ∼ 𝒩 ⁡ ( 0 , Σ ) ​ [ f ⁡ ( c + Z ) ] \bar{f}(c)=\mathbb{E}_{Z\sim\mathcal{N}(0,\Sigma)}[f(c+Z)] . By Assumption B.2 , a ∼ π ϕ ∗ ​ ( a | s ) a\sim\pi_{\phi^{*}}(a|s) can be written as a = c + Z a=c+Z with Z ∼ 𝒩 ⁡ ( 0 , Σ ) Z\sim\mathcal{N}(0,\Sigma) . By Lemma B.6 and, applied to f ¯ \bar{f} , which we know fulfills Assumption B.3 , we obtain on B ⁡ ( c , D ) B(c,D) the gradient bound ‖ ∇ f ¯ ​ ( c ) ‖ ≤ L ^ D + β 2 ​ D . \|\nabla\bar{f}(c)\|\leq\frac{\widehat{L}}{D}+\frac{\beta}{2}D\,. From Lemma B.7 , we know that the gradient norm of f f can then be bounded as ‖ ∇ f ​ ( c ) ‖ ≤ L ^ D + β 2 ​ D + β ​ 𝔼 ​ [ ‖ Z ‖ ] ⏟ ≔ G . \|\nabla f(c)\|\leq\underbrace{\frac{\widehat{L}}{D}+\frac{\beta}{2}D+\beta\,\mathbb{E}[\|Z\|]}_{\coloneq G}\,. We then know from Lemma B.8 , that in B ⁡ ( c , r ) B(c,r) with non-violating radius r = 1 β ​ ( K δ − G ) r=\frac{1}{\beta}(\frac{K}{\delta}-G) there can be no action pairs a 1 , a 2 a_{1},a_{2} with ‖ a 1 − a 2 ‖ ≤ δ \|a_{1}-a_{2}\|\leq\delta and | R ~ ​ ( s , a 1 ) − R ~ ​ ( s , a 2 ) | > K |\widetilde{R}(s,a_{1})-\widetilde{R}(s,a_{2})|>K , thus such violations can only occur if at least one action is outside B ⁡ ( c , r ) B(c,r) : P ⁡ ( ‖ a 1 − a 2 ‖ ≤ δ , | f ⁡ ( a 1 ) − f ⁡ ( a 2 ) | > K ) ≤ 2 ​ P ​ ( ‖ a 1 − c ‖ > r ) . P(\|a_{1}-a_{2}\|\leq\delta,\ |f(a_{1})-f(a_{2})|>K)\leq 2\,P(\|a_{1}-c\|>r)\,. Finally, a 1 − c ∼ 𝒩 ⁡ ( 0 , Σ ) a_{1}-c\sim\mathcal{N}(0,\Sigma) , so P ⁡ ( ‖ a 1 − c ‖ > r ) = P ⁡ ( ‖ Z ‖ > r ) P(\|a_{1}-c\|>r)=P(\|Z\|>r) . ∎

### B.2 BT-Loss Lower Bound Based on Sharpness

Now that we have connected D − L ^ D-\widehat{L} robustness to the probability of actions pairs violating a flatness assumption P ⁡ ( S K , δ | π ϕ ) P(S_{K,\delta}|\pi_{\phi}) , we now analyze the incurred excess BT loss ℒ BT \mathcal{L}_{\mathrm{BT}} due to these violations. For this purpose, we make the assumption of a Lipschitz continuous true reward (Assumption B.4 )

###### Proposition B.10 .

For a prompt s ∼ P ⁡ ( s ) s\sim P(s) , a pair of actions ( a 1 , a 2 ) (a_{1},a_{2}) , L L -Lipschitz true reward function R ∗ R^{*} , reward model R ~ \widetilde{R} , the excess-risk can be lower bounded as ℒ BT ​ ( R ~ ) − ℒ BT ​ ( R ∗ ) ≥ 2 ​ ( σ ⁡ ( K ) − σ ⁡ ( L ​ δ ) ) 2 ​ P ​ ( S K , δ ) , \mathcal{L}_{\mathrm{BT}}(\widetilde{R})-\mathcal{L}_{\mathrm{BT}}(R^{*})\geq 2(\sigma(K)-\sigma(L\delta))^{2}P(S_{K,\delta})\,, (17) where S K , δ := { ( s , a 1 , a 2 ) : ∥ a 1 − a 2 ∥ ≤ δ , | R ~ ( s , a 1 ) − R ~ ( s , a 2 ) | > K } S_{K,\delta}:=\left\{(s,a_{1},a_{2}):\|a_{1}-a_{2}\|\leq\delta,|\widetilde{R}(s,a_{1})-\widetilde{R}(s,a_{2})|>K\right\} is the set of action pairs for which the RM R ~ \widetilde{R} is not K K -Lipschitz continuous and Pr ⁡ ( S K , δ ) \Pr(S_{K,\delta}) is the probability of an action pair sampled ( a 1 , a 2 ) ∼ π ϕ ( ⋅ | s ) (a_{1},a_{2})\sim\pi_{\phi}(\cdot|s) being in this set.