It Takes Two: Your GRPO Is Secretly DPO — part 3/5
Cleaned-text code points [30602, 45755). Bibliography omitted here; source.md includes it.
RL post-training has demonstrated significant success in enhancing LLM performance across diverse domains [ 39 , 43 ] . Unlike SFT, RL requires the model to generate online samples during training. Although modern frameworks integrate high-throughput inference engines such as vLLM and SGLang, the autoregressive nature of LLMs ensures that the generation phase remains a primary computational bottleneck. This challenge is exacerbated by the common intuition that LLM-based RL often necessitates large group sizes to achieve good performance. To mitigate this overhead, recent studies have proposed selective or adaptive sampling techniques to reduce the number of rollouts without compromising performance [ 47 , 44 , 49 ] . Within this context, 2-GRPO serves as a robust baseline. Furthermore, our contrastive analysis of GRPO opens a new design space for developing efficient sampling algorithms in RLVR.
## Appendix B Theorems
### B.1 Variance Reduction of Policy Gradient Estimate
Given a prompt, consider the random variable (r.v.) of the reward r r (which can be replaced by the advantage a a ) and the r.v. of the policy gradient 𝒈 \bm{g} (corresponding to 1 | o i | ∑ t = 0 | o i | ∇ θ log π θ ( o i , t | o i , < t , q ) \frac{1}{|o_{i}|}\sum_{t=0}^{|o_{i}|}\nabla_{\theta}\log\pi_{\theta}(o_{i,t}|o_{i,<t},q) in VPG or 1 | o i | ∑ t | o i | ∇ θ ρ i , t \frac{1}{|o_{i}|}\sum_{t}^{|o_{i}|}\nabla_{\theta}\rho_{i,t} in PPO/GRPO). Since 𝔼 [ 𝒈 ] = 0 {\mathbb{E}}[\bm{g}]=0 over all potential actions, the variance of the product of these r.v.’s can be written as: Var ( r ⋅ 𝒈 ) \displaystyle\text{Var}(r\cdot\bm{g}) = Var ( 𝒈 ) [ Var ( r ) + ( 𝔼 [ r ] ) 2 ] + Cov ( r 2 , 𝒈 2 ) − ( Cov ( r , 𝒈 ) ) 2 ⏟ Interaction term . \displaystyle=\text{Var}(\bm{g})\left[\text{Var}(r)+({\mathbb{E}}[r])^{2}\right]+\underbrace{\text{Cov}(r^{2},\bm{g}^{2})-(\text{Cov}(r,\bm{g}))^{2}}_{\text{Interaction term}}\,. (11) The interaction term can be ignored when importance sampling and clipping are applied, as the gradient is bounded in a small region. Previous work [ 3 , 31 , 32 , 33 ] shows that replacing raw rewards with advantage functions ( 𝔼 [ a ] = 0 {\mathbb{E}}[a]=0 ) effectively reduces variance, leading to more stable and improved RL optimization.
### B.2 Mean Estimation with Samples n = 2 {n=2}
The instability of normalization with extremely small samples is a well-documented phenomenon in classical statistics, dating back to the seminal work of William Sealy Gosset (published under the pen name Student ) [ 36 ] . For a sample size of n = 2 n=2 , the degrees of freedom d f = 1 df=1 result in a normalization factor that follows a Cauchy distribution. Such small-sample estimates of variance are highly skewed, leading to normalized outputs with infinite variance and no defined mean, undermining the goal of statistical stability.
### B.3 Reveal GRPO as Contrastive Learning
###### Proof of Proposition 3.2 .
In the RLVR setting, rewards are binary, which leads to binary advantages given a prompt. Let A q + , A q − A^{+}_{q},A^{-}_{q} denote the positive and negative advantage, respectively. From Eq. ( 4 ), we can have A q + \displaystyle A^{+}_{q} = 1 − p ^ q p ^ q ( 1 − p ^ q ) = 1 − p ^ q p ^ q , \displaystyle=\frac{1-\hat{p}_{q}}{\sqrt{\hat{p}_{q}(1-\hat{p}_{q})}}=\sqrt{\frac{1-\hat{p}_{q}}{\hat{p}_{q}}}\;, (12) A q − \displaystyle A^{-}_{q} = 0 − p ^ q p ^ q ( 1 − p ^ q ) = − p ^ q 1 − p ^ q . \displaystyle=\frac{0-\hat{p}_{q}}{\sqrt{\hat{p}_{q}(1-\hat{p}_{q})}}=-\sqrt{\frac{\hat{p}_{q}}{1-\hat{p}_{q}}}\;.
The clipping function is clip ( x , 1 − ϵ , 1 + ϵ ) = { x , | x − 1 | ≤ ϵ 1 − ϵ , x < 1 − ϵ 1 + ϵ , x > 1 + ϵ , \text{clip}(x,1-\epsilon,1+\epsilon)=\begin{cases}x,&|x-1|\leq\epsilon\\ 1-\epsilon,&x<1-\epsilon\\ 1+\epsilon,&x>1+\epsilon\end{cases}\,, (13) which means that x x will be assigned to 1 − ϵ 1-\epsilon ( 1 + ϵ 1+\epsilon ) if x x is less (greater) than 1 − ϵ 1-\epsilon ( 1 + ϵ 1+\epsilon ). For simplifying notation, let 𝒞 ϵ + ( x ) = min [ x , 1 + ϵ ] \mathcal{C}_{\epsilon}^{+}(x)=\min[x,1+\epsilon] and 𝒞 ϵ − = max [ x , 1 − ϵ ] \mathcal{C}_{\epsilon}^{-}=\max[x,1-\epsilon] .
The key derivation of rewriting GRPO objective is as follows: 𝒥 GRPO ( θ ) \displaystyle\mathcal{J}_{\text{GRPO}}(\theta) (14) = 𝔼 q ∼ 𝒬 { o i } i = 1 G ∼ π θ old ( ⋅ | q ) 1 G ∑ i = 1 G 1 | o i | ∑ t = 1 | o i | 𝒞 ϵ ( π θ ( o i , t | o i , < t , q ) π θ old ( o i , t | o i , < t , q ) A i , t ) , \displaystyle=\mathbb{E}_{\begin{subarray}{c}q\sim\mathcal{Q}\\ \{o_{i}\}_{i=1}^{G}\sim\pi_{\theta_{\text{old}}}(\cdot|q)\end{subarray}}\frac{1}{G}\sum_{i=1}^{G}\frac{1}{|o_{i}|}\sum_{t=1}^{|o_{i}|}\mathcal{C}_{\epsilon}\left(\frac{\pi_{\theta}(o_{i,t}|o_{i,<t},q)}{\pi_{\theta_{\text{old}}}(o_{i,t}|o_{i,<t},q)}A_{i,t}\right)\,, = 𝔼 q ∼ 𝒬 { o j } j = 1 G + ∼ π θ old + ( ⋅ | q ) { o k } k = 1 G − ∼ π θ old − ( ⋅ | q ) \displaystyle=\mathbb{E}_{\begin{subarray}{c}q\sim\mathcal{Q}\\ \{o_{j}\}_{j=1}^{G^{+}}\sim\pi_{\theta_{\text{old}}}^{+}(\cdot|q)\\ \{o_{k}\}_{k=1}^{G^{-}}\sim\pi_{\theta_{\text{old}}}^{-}(\cdot|q)\end{subarray}} 1 G ( ∑ j = 1 G + 1 | o j | ∑ t = 1 | o j | A j + 𝒞 ϵ + ( π θ ( o j , t | o j , < t , q ) π θ old ( o j , t | o j , < t , q ) ) + ∑ k = 1 G − 1 | o k | ∑ t = 1 | o k | A k − 𝒞 ϵ − ( π θ ( o k , t | o k , < t , q ) π θ old ( o k , t | o k , < t , q ) ) ) , \displaystyle\frac{1}{G}\left(\sum_{j=1}^{G^{+}}\frac{1}{|o_{j}|}\sum_{t=1}^{|o_{j}|}A_{j}^{+}\mathcal{C}_{\epsilon}^{+}\left(\frac{\pi_{\theta}(o_{j,t}|o_{j,<t},q)}{\pi_{\theta_{\text{old}}}(o_{j,t}|o_{j,<t},q)}\right)+\sum_{k=1}^{G^{-}}\frac{1}{|o_{k}|}\sum_{t=1}^{|o_{k}|}A_{k}^{-}\mathcal{C}_{\epsilon}^{-}\left(\frac{\pi_{\theta}(o_{k,t}|o_{k,<t},q)}{\pi_{\theta_{\text{old}}}(o_{k,t}|o_{k,<t},q)}\right)\right)\,, = 𝔼 q ∼ 𝒬 { o j } j = 1 G + ∼ π θ old + ( ⋅ | q ) { o k } k = 1 G − ∼ π θ old − ( ⋅ | q ) \displaystyle=\mathbb{E}_{\begin{subarray}{c}q\sim\mathcal{Q}\\ \{o_{j}\}_{j=1}^{G^{+}}\sim\pi_{\theta_{\text{old}}}^{+}(\cdot|q)\\ \{o_{k}\}_{k=1}^{G^{-}}\sim\pi_{\theta_{\text{old}}}^{-}(\cdot|q)\end{subarray}} A q + G + G 1 G + ∑ j = 1 G + 1 | o j | ∑ t = 1 | o j | 𝒞 ϵ + ( π θ ( o j , t | o j , < t , q ) π θ old ( o j , t | o j , < t , q ) ) + A q − G − G 1 G − ∑ k = 1 G − 1 | o k | ∑ t = 1 | o k | 𝒞 ϵ − ( π θ ( o k , t | o k , < t , q ) π θ old ( o k , t | o k , < t , q ) ) , \displaystyle A^{+}_{q}\frac{G^{+}}{G}\frac{1}{G^{+}}\sum_{j=1}^{G^{+}}\frac{1}{|o_{j}|}\sum_{t=1}^{|o_{j}|}\mathcal{C}_{\epsilon}^{+}\left(\frac{\pi_{\theta}(o_{j,t}|o_{j,<t},q)}{\pi_{\theta_{\text{old}}}(o_{j,t}|o_{j,<t},q)}\right)+A^{-}_{q}\frac{G^{-}}{G}\frac{1}{G^{-}}\sum_{k=1}^{G^{-}}\frac{1}{|o_{k}|}\sum_{t=1}^{|o_{k}|}\mathcal{C}_{\epsilon}^{-}\left(\frac{\pi_{\theta}(o_{k,t}|o_{k,<t},q)}{\pi_{\theta_{\text{old}}}(o_{k,t}|o_{k,<t},q)}\right)\,, = 𝔼 q ∼ 𝒬 { o j } j = 1 G + ∼ π θ old + ( ⋅ | q ) { o k } k = 1 G − ∼ π θ old − ( ⋅ | q ) \displaystyle=\mathbb{E}_{\begin{subarray}{c}q\sim\mathcal{Q}\\ \{o_{j}\}_{j=1}^{G^{+}}\sim\pi_{\theta_{\text{old}}}^{+}(\cdot|q)\\ \{o_{k}\}_{k=1}^{G^{-}}\sim\pi_{\theta_{\text{old}}}^{-}(\cdot|q)\end{subarray}} Var ^ G ( q ) ( 1 G + ∑ j = 1 G + 1 | o j | ∑ t = 1 | o j | 𝒞 ϵ + ( π θ ( o j , t | o j , < t , q ) π θ old ( o j , t | o j , < t , q ) ) − 1 G − ∑ k = 1 G − 1 | o k | ∑ t = 1 | o k | 𝒞 ϵ − ( π θ ( o k , t | o k , < t , q ) π θ old ( o k , t | o k , < t , q ) ) ) . \displaystyle\sqrt{\widehat{\mathrm{Var}}_{G}(q)}\left(\frac{1}{G^{+}}\sum_{j=1}^{G^{+}}\frac{1}{|o_{j}|}\sum_{t=1}^{|o_{j}|}\mathcal{C}_{\epsilon}^{+}\left(\frac{\pi_{\theta}(o_{j,t}|o_{j,<t},q)}{\pi_{\theta_{\text{old}}}(o_{j,t}|o_{j,<t},q)}\right)-\frac{1}{G^{-}}\sum_{k=1}^{G^{-}}\frac{1}{|o_{k}|}\sum_{t=1}^{|o_{k}|}\mathcal{C}_{\epsilon}^{-}\left(\frac{\pi_{\theta}(o_{k,t}|o_{k,<t},q)}{\pi_{\theta_{\text{old}}}(o_{k,t}|o_{k,<t},q)}\right)\right)\,. The second equation is obtained by dividing the trajectories into two groups: positive and negative. The third equation is obtained by the fact that all positive advantages are the same and that all negative advantages are the same. Since A + G + G = 1 − p ^ p ^ p ^ = ( 1 − p ^ ) p ^ A^{+}\frac{G^{+}}{G}=\sqrt{\frac{1-\hat{p}}{\hat{p}}}\hat{p}=\sqrt{(1-\hat{p})\hat{p}} and A − G − G = − ( 1 − p ^ ) p ^ A^{-}\frac{G^{-}}{G}=-\sqrt{(1-\hat{p})\hat{p}} , we obtain Eq. ( 7 ). When G → ∞ G\to\infty , we have the following facts: lim G → ∞ G + = ∞ , \displaystyle\lim_{G\to\infty}G^{+}=\infty\;, (15) lim G → ∞ G − = ∞ \displaystyle\lim_{G\to\infty}G^{-}=\infty lim G → ∞ ( 1 − p ^ ) p ^ = ( 1 − p ) p , \displaystyle\lim_{G\to\infty}\sqrt{(1-\hat{p})\hat{p}}=\sqrt{(1-p)p}\;, lim G + → ∞ 1 G + ∑ j = 1 G + f ( o j ) = 𝔼 o j ∼ O θ + f ( o j ) , \displaystyle\lim_{G^{+}\to\infty}\frac{1}{G^{+}}\sum_{j=1}^{G^{+}}f(o_{j})={\mathbb{E}}_{o_{j}\sim O_{\theta}^{+}}f(o_{j})\;, lim G − → ∞ 1 G − ∑ k = 1 G − f ( o k ) = 𝔼 o k ∼ O θ − f ( o k ) . \displaystyle\lim_{G^{-}\to\infty}\frac{1}{G^{-}}\sum_{k=1}^{G^{-}}f(o_{k})={\mathbb{E}}_{o_{k}\sim O_{\theta}^{-}}f(o_{k})\;.
Then the GRPO objective has the following gradient w.r.t. parameter θ \theta : ∇ θ 𝒥 GRPO = \displaystyle\nabla_{\theta}\mathcal{J}_{\text{GRPO}}= 𝔼 q ∼ 𝒬 Var ^ G ( q ) ( 1 G q + ∑ j = 1 G q + ∑ t | o j + | 𝟙 j , t ϵ ∇ θ π θ ( o j , t + | o j , < t + , q ) | o j + | π θ old ( o j , t + | o j , < t + , q ) − 1 G q − ∑ k = 1 G q − ∑ t o k − 𝟙 k , t ϵ ∇ θ π θ ( o k , t − | o k , < t − , q ) | o k − | π θ old ( o k , t − | o k , < t − , q ) ) \displaystyle\underset{q\sim\mathcal{Q}}{{\mathbb{E}}}\sqrt{\widehat{\mathrm{Var}}_{G}(q)}\Bigg(\frac{1}{G_{q}^{+}}\sum_{j=1}^{G_{q}^{+}}\sum_{t}^{|o_{j}^{+}|}\frac{\mathbbm{1}^{\epsilon}_{j,t}\nabla_{\theta}\pi_{\theta}(o_{j,t}^{+}|o_{j,<t}^{+},q)}{|o_{j}^{+}|\pi_{\theta_{\text{old}}}(o_{j,t}^{+}|o_{j,<t}^{+},q)}-\frac{1}{G_{q}^{-}}\sum_{k=1}^{G_{q}^{-}}\sum_{t}^{o_{k}^{-}}\frac{\mathbbm{1}^{\epsilon}_{k,t}\nabla_{\theta}\pi_{\theta}(o_{k,t}^{-}|o_{k,<t}^{-},q)}{|o_{k}^{-}|\pi_{\theta_{\text{old}}}(o_{k,t}^{-}|o_{k,<t}^{-},q)}\Bigg) (16) = 𝔼 q ∼ 𝒬 [ 1 G q + ∑ j = 1 G q + ∑ t | o j + | c ( o j , t + | o j , < t + , q ) ∇ θ π θ ( o j , t + | o j , < t + , q ) ⏟ Positive − 1 G q − ∑ k = 1 G q − ∑ t | o k − | c ( o k , t − | o k , < t − , q ) ∇ θ π θ ( o k , t − | o k , < t − , q ) ⏟ Negative ] \displaystyle=\underset{q\sim\mathcal{Q}}{{\mathbb{E}}}\Bigg[\underbrace{\frac{1}{G_{q}^{+}}\sum_{j=1}^{G_{q}^{+}}\sum_{t}^{|o_{j}^{+}|}c(o_{j,t}^{+}|o_{j,<t}^{+},q)\nabla_{\theta}\pi_{\theta}(o_{j,t}^{+}|o_{j,<t}^{+},q)}_{\text{Positive}}-\underbrace{\frac{1}{G_{q}^{-}}\sum_{k=1}^{G_{q}^{-}}\sum_{t}^{|o_{k}^{-}|}c(o_{k,t}^{-}|o_{k,<t}^{-},q)\nabla_{\theta}\pi_{\theta}(o_{k,t}^{-}|o_{k,<t}^{-},q)}_{\text{Negative}}\Bigg] (17) where 𝟙 j , t ϵ \mathbbm{1}^{\epsilon}_{j,t} is an indicator function if the token o j , t o_{j,t} is clipped and c ( o i , t | o i , < t , q ) := Var ^ ( q ) 𝟙 i , t ϵ | o i | π θ old ( o i , t | o i , < t , q ) c(o_{i,t}|o_{i,<t},q):=\frac{\sqrt{\widehat{\mathrm{Var}}(q)}\mathbbm{1}^{\epsilon}_{i,t}}{|o_{i}|\pi_{\theta_{\text{old}}}(o_{i,t}|o_{i,<t},q)} . Compare Eq. ( 17 ) with Def. 3.1 , the derivative of GRPO is a Monte Carlo estimator of contrastive derivative. ∎
### B.4 Further Discussion on Importance Sampling and the Log-likelihood Term
Most autoregressive LLMs adopt causal probability modelling as log π θ ( o | q ) = ∑ log π θ ( o t | o < t , q ) \log\pi_{\theta}(o|q)=\sum\log\pi_{\theta}(o_{t}|o_{<t},q) . This decomposition leads to the following trajectory-level form to describe the gradient of token probabilities: ∇ θ log π θ ( o | q ) = ∑ t 1 π θ ( o t | o < t , q ) ∇ θ π θ ( o t | o < t , q ) . \nabla_{\theta}\log\pi_{\theta}(o|q)=\sum_{t}\frac{1}{\pi_{\theta}(o_{t}|o_{<t},q)}\nabla_{\theta}\pi_{\theta}(o_{t}|o_{<t},q)\,. (18) DPO follows a similar structural derivation.
It is worth mentioning that the importance sampling in PPO can be viewed as a natural extension of such gradient form for online on/off-policy RL [ 33 ] . However, the token-level importance sampling in PPO and vanilla GRPO often obscures this direct connection at the trajectory level.
Recent subsequent variants of GRPO [ 46 , 45 , 28 ] , e.g., GSPO and TIC-GRPO, utilize sequence-level importance sampling. This formulation allows us to draw a direct connection between importance sampling and the log-likelihood terms: ∇ θ π θ ( o ∣ q ) π θ old ( o ∣ q ) = π θ ( o ∣ q ) π θ old ( o ∣ q ) ∑ t 1 π θ ( o t ∣ o < t , q ) ∇ θ π θ ( o t ∣ o < t , q ) . \nabla_{\theta}\frac{\pi_{\theta}(o\mid q)}{\pi_{\theta_{\mathrm{old}}}(o\mid q)}=\frac{\pi_{\theta}(o\mid q)}{\pi_{\theta_{\mathrm{old}}}(o\mid q)}\sum_{t}\frac{1}{\pi_{\theta}(o_{t}\mid o_{<t},q)}\nabla_{\theta}\pi_{\theta}(o_{t}\mid o_{<t},q)\,. (19) It is straightforward to see from the gradient form that the importance sampling term adjusts the Log-likelihood term by a coefficient π θ ( o ∣ q ) π θ old ( o | q ) \frac{\pi_{\theta}(o\mid q)}{\pi_{\theta_{\mathrm{old}}}(o|q)} . The token-level importance sampling in PPO and GRPO behaves similarly by applying token-level correction.
The clipping applied on top of importance sampling is a minor additional modification, which we do not elaborate on here.
### B.5 Proof of Lemma B.1 : DPO is 1-vs-1 contrastive learning
###### Lemma B.1 .
The DPO loss is a 1 1 -vs- 1 1 contrastive loss estimator.
###### Proof of Lemma B.1 .
The DPO loss (Eq. ( 6 )) has the following derivatives: \displaystyle ∇ θ ℒ DPO = − β 𝔼 [ ( q , o + , o − ) ∼ 𝒟 DPO ] [ σ ( r ^ θ ( q , o − ) − r ^ θ ( q , o + ) ) ( ∇ θ log π θ ( o + | q ) − ∇ θ log π θ ( o − | q ) ) ] \displaystyle\nabla_{\theta}\mathcal{L}_{\text{DPO}}=-\beta\underset{[(q,o^{+},o^{-})\sim{\mathcal{D}}_{\text{DPO}}]}{{\mathbb{E}}}\Big[\sigma(\hat{r}_{\theta}(q,o^{-})-\hat{r}_{\theta}(q,o^{+}))\left(\nabla_{\theta}\log\pi_{\theta}(o^{+}|q)-\nabla_{\theta}\log\pi_{\theta}(o^{-}|q)\right)\Big]\;\, (20) = − 𝔼 [ ( q , o + , o − ) ∼ 𝒟 DPO ] [ ∑ t | o + | c ( o t + | o < t + , q ) ∇ θ π θ ( o t + | o < t + , q ) ⏟ Positive − ∑ t | o − | c ( o t − | o < t − , q ) ∇ θ π θ ( o t − | o < t − , q ) ⏟ Negative ] \displaystyle=-\underset{[(q,o^{+},o^{-})\sim{\mathcal{D}}_{\text{DPO}}]}{{\mathbb{E}}}\Bigg[\underbrace{\sum_{t}^{|o^{+}|}c(o^{+}_{t}|o^{+}_{<t},q)\nabla_{\theta}\pi_{\theta}(o^{+}_{t}|o^{+}_{<t},q)}_{\text{Positive}}-\underbrace{\sum_{t}^{|o^{-}|}c(o^{-}_{t}|o^{-}_{<t},q)\nabla_{\theta}\pi_{\theta}(o^{-}_{t}|o^{-}_{<t},q)}_{\text{Negative}}\Bigg] where r ^ θ = β ( x , y ) log π θ ( y | x ) π ref ( y | x ) \hat{r}_{\theta}=\beta(x,y)\log\frac{\pi_{\theta}(y|x)}{\pi_{\text{ref}}(y|x)} ; σ \sigma denotes the sigmoid function; and c ( o t | o < t , q ) := β σ ( r ^ θ ( q , o − ) − r ^ θ ( q , o + ) ) π θ ( o | q ) c(o_{t}|o_{<t},q):=\frac{\beta\sigma(\hat{r}_{\theta}(q,o^{-})-\hat{r}_{\theta}(q,o^{+}))}{\pi_{\theta}(o|q)} aligning with Def. 3.1 . ∎
### B.6 GRPO v.s. DPO from Contrastive Learning