It Takes Two: Your GRPO Is Secretly DPO β part 4/5
Cleaned-text code points [45755, 61698). Bibliography omitted here; source.md includes it.
As shown in Appx. B.3 and B.5 , GRPO and DPO admit the following gradient formulations: β ΞΈ π₯ GRPO = πΌ q βΌ π¬ β [ β t | o + | c β‘ ( o t + | o < t + , q ) β β ΞΈ Ο ΞΈ β ( o t + | o < t + , q ) β Positive β β t | o k β | c β‘ ( o k , t β | o k , < t β , q ) β β ΞΈ Ο ΞΈ β ( o k , t β | o k , < t β , q ) β Negative ] \displaystyle\nabla_{\theta}\mathcal{J}_{\text{GRPO}}=\underset{q\sim\mathcal{Q}}{\mathbb{E}}\Bigg[\underbrace{\sum_{t}^{|o^{+}|}c(o_{t}^{+}|o_{<t}^{+},q)\nabla_{\theta}\pi_{\theta}(o_{t}^{+}|o_{<t}^{+},q)}_{\text{Positive}}-\underbrace{\sum_{t}^{|o_{k}^{-}|}c(o_{k,t}^{-}|o_{k,<t}^{-},q)\nabla_{\theta}\pi_{\theta}(o_{k,t}^{-}|o_{k,<t}^{-},q)}_{\text{Negative}}\Bigg] (21) β ΞΈ β DPO = β πΌ [ ( q , o + , o β ) βΌ π DPO ] β [ β t | o + | c β‘ ( o t + | o < t + , q ) β β ΞΈ Ο ΞΈ β ( o t + | o < t + , q ) β Positive β β t | o β | c β‘ ( o t β | o < t β , q ) β β ΞΈ Ο ΞΈ β ( o t β | o < t β , q ) β Negative ] \displaystyle\nabla_{\theta}\mathcal{L}_{\text{DPO}}=-\underset{[(q,o^{+},o^{-})\sim\mathcal{D}_{\text{DPO}}]}{\mathbb{E}}\Bigg[\underbrace{\sum_{t}^{|o^{+}|}c(o^{+}_{t}|o^{+}_{<t},q)\nabla_{\theta}\pi_{\theta}(o^{+}_{t}|o^{+}_{<t},q)}_{\text{Positive}}-\underbrace{\sum_{t}^{|o^{-}|}c(o^{-}_{t}|o^{-}_{<t},q)\nabla_{\theta}\pi_{\theta}(o^{-}_{t}|o^{-}_{<t},q)}_{\text{Negative}}\Bigg] (22)
These expressions reveal that both maximizing the GRPO objective and minimizing the DPO loss correspond to the same underlying contrastive learning mechanism, differing only in the specific design of the coefficient c β‘ ( o t + | o < t + , q ) c(o_{t}^{+}|o_{<t}^{+},q) .
The key distinction lies in how the coefficient c β‘ ( β
) c(\cdot) is instantiated under different RL settings (online vs. offline): β’ Importance sampling term (online GRPO) vs. log-likelihood term (offline DPO) (see Appx. B.4 for detailed discussion).
β’ Reference-model regularization : an explicit KL term (GRPO) vs. implicit incorporation into the βadvantageβ (DPO).
Importantly, these differences do not alter the fundamental optimization structure, but rather reflect distinct design choices tailored to their respective RL regimes.
### B.7 Proof of Proposition 4.1
###### Proof.
Var β‘ ( π + β c β π β ) \displaystyle\mathrm{Var}(\bm{g}^{+}-c\bm{g}^{-}) = Var β‘ ( π + ) + c 2 β Var β ( π β ) β 2 β c β Cov β ( π + , π β ) , \displaystyle=\mathrm{Var}(\bm{g}^{+})+c^{2}\mathrm{Var}(\bm{g}^{-})-2c\mathrm{Cov}(\bm{g}^{+},\bm{g}^{-})\,, (23) = Var β‘ ( π + ) β Cov 2 β ( π + , π β ) Var β‘ ( π β ) , \displaystyle=\mathrm{Var}(\bm{g}^{+})-\frac{\mathrm{Cov}^{2}(\bm{g}^{+},\bm{g}^{-})}{\mathrm{Var}(\bm{g}^{-})}\,, = ( 1 β Ο 2 ) β Var β ( π + ) . \displaystyle=(1-\rho^{2})\mathrm{Var}(\bm{g}^{+})\,. The first equation is obtained by the definition of variance. The second equation is obtained by substituting c = Cov β‘ ( π + , π β ) Var β‘ ( π β ) c=\frac{\mathrm{Cov}(\bm{g}^{+},\bm{g}^{-})}{\mathrm{Var}(\bm{g}^{-})} . The third equation is hold because Ο = Cov β‘ ( π + , π β ) Var β‘ ( π + ) β Var β ( π β ) \rho=\frac{\mathrm{Cov}(\bm{g}^{+},\bm{g}^{-})}{\sqrt{\mathrm{Var}(\bm{g}^{+})\mathrm{Var}(\bm{g}^{-})}} . On the other hand, consider f β‘ ( c ) = c 2 β Var β ( π β ) β 2 β c β Cov β ( π + , π β ) f(c)=c^{2}\mathrm{Var}(\bm{g}^{-})-2c\mathrm{Cov}(\bm{g}^{+},\bm{g}^{-}) . If 0 β€ c β€ 2 β Cov 2 β ( π + , π β ) Var β‘ ( π β ) 0\leq c\leq 2\frac{\mathrm{Cov}^{2}(\bm{g}^{+},\bm{g}^{-})}{\mathrm{Var}(\bm{g}^{-})} , then f β‘ ( c ) β€ 0 f(c)\leq 0 . β
### B.8 The Correlation between The Positive and The Negative
We do not have access to the joint distribution of positive and negative gradients, so direct empirical estimation of their covariance is infeasible. Instead, we use the law of total covariance: Cov β‘ ( π + , π β ) = πΌ p β [ Cov β‘ ( π + , π β β£ p ) ] + Cov p β ( πΌ β‘ [ π + | p ] , πΌ β‘ [ π β | p ] ) . \mathrm{Cov}(\bm{g}^{+},\bm{g}^{-})=\mathbb{E}_{p}[\mathrm{Cov}(\bm{g}^{+},\bm{g}^{-}\mid p)]+\mathrm{Cov}_{p}(\mathbb{E}[\bm{g}^{+}|p],\mathbb{E}[\bm{g}^{-}|p])\,. (24) Under conditionally independent sampling, the first term is zero, so we estimate the second term across prompts. Statistics are computed over 100 prompts with 10 responses each on MATH with Qwen-1.5B. As high-dimensional vectors usually have small dot-products, we report a baseline as reference where pos/neg pairs are randomly permuted.
As shown in the table, the covariance between positive and negative gradients from the same prompt is significantly larger than that between randomly paired positive and negative gradients, which confirms our assumption.
### B.9 Proof of Proposition C.1
###### Proof.
Case 1. Notice that Ο ^ = 1 2 β N β β k = 1 2 β N ( X k β ΞΌ ^ ) 2 = ΞΌ ^ β ( 1 β ΞΌ ^ ) \hat{\sigma}=\sqrt{\frac{1}{2N}\sum_{k=1}^{2N}(X_{k}-\hat{\mu})^{2}}=\sqrt{\hat{\mu}(1-\hat{\mu})} and ΞΌ ^ = 1 2 β N β β k = 1 2 β N X k \hat{\mu}=\frac{1}{2N}\sum_{k=1}^{2N}X_{k} . Fix an index i i and condition on the event { X i = x } \{X_{i}=x\} with x β { 0 , 1 } x\in\{0,1\} . In this case, by the strong law of large numbers and the continuous mapping theorem, we have ΞΌ ^ β a . s . p \hat{\mu}\stackrel{{\scriptstyle a.s.}}{{\rightarrow}}p and Ο ^ β a . s . p β‘ ( 1 β p ) \hat{\sigma}\stackrel{{\scriptstyle a.s.}}{{\rightarrow}}\sqrt{p(1-p)} . Thus, it follows that lim Ο΅ β 0 lim N β β πΌ β‘ [ Y i β£ X i = x ] = x β p p β‘ ( 1 β p ) . \lim_{\epsilon\rightarrow 0}\,\lim_{N\to\infty}\,{\mathbb{E}}\!\left[Y_{i}\mid X_{i}=x\right]=\frac{x-p}{\sqrt{p(1-p)}}.
Case 2. When X i , 1 = X i , 2 X_{i,1}=X_{i,2} , we have X i , j = ΞΌ ^ i X_{i,j}=\hat{\mu}_{i} and Y i , j = 0 Y_{i,j}=0 for any j β { 1 , 2 } j\in\{1,2\} . When X i , 1 β X i , 2 X_{i,1}\neq X_{i,2} , we have ΞΌ ^ i = 0.5 \hat{\mu}_{i}=0.5 , Ο ^ i = 0.5 \hat{\sigma}_{i}=0.5 , and Y i , j = 2 β X i , j β 1 1 + 2 β Ο΅ . Y_{i,j}=\frac{2X_{i,j}-1}{1+2\epsilon}. By the law of total expectation, it follows that πΌ β‘ [ Y i , j β£ X i , j = 1 ] = 1 β p 1 + 2 β Ο΅ , πΌ β‘ [ Y i , j β£ X i , j = 0 ] = β p 1 + 2 β Ο΅ . {\mathbb{E}}\left[Y_{i,j}\mid X_{i,j}=1\right]=\frac{1-p}{1+2\epsilon},\qquad{\mathbb{E}}\left[Y_{i,j}\mid X_{i,j}=0\right]=\frac{-p}{1+2\epsilon}. Thus, we have lim Ο΅ β 0 πΌ β‘ [ Y i , j β£ X i , j = x ] = x β p . \lim_{\epsilon\rightarrow 0}{\mathbb{E}}\!\left[Y_{i,j}\mid X_{i,j}=x\right]=x-p. β
### B.10 Proof of Lemma C.3
###### Proof of Lemma C.3 .
Var β‘ ( π ^ B ) \displaystyle\mathrm{Var}(\hat{\bm{g}}_{B}) = Var { π i } i = 1 B β ( 1 B β β i = 1 B π β‘ ( π i ) ) \displaystyle=\mathrm{Var}_{\{{\bm{x}}_{i}\}_{i=1}^{B}}\left(\frac{1}{B}\sum_{i=1}^{B}\bm{g}({\bm{x}}_{i})\right) (25) = 1 B 2 β β i = 1 B Var π i β ( π β‘ ( π i ) ) = Var π β ( π β ( π ) ) B . \displaystyle=\frac{1}{B^{2}}\sum_{i=1}^{B}\mathrm{Var}_{{\bm{x}}_{i}}\!\left(\bm{g}({\bm{x}}_{i})\right)=\frac{\mathrm{Var}_{{\bm{x}}}\!\left(\bm{g}({\bm{x}})\right)}{B}\;.
where the second and third equalities are obtained by the properties of independence and identity in i.i.d. data, respectively. By the above equation, increasing B B decreases Var \mathrm{Var} . β
## Appendix C Theoretical Analysis of 2-GRPO
### C.1 Implicit Weighting in Stochastic Optimization
At first glance, 2-GRPO appears to use only fixed advantages, A + = 1 A^{+}=1 and A β = β 1 A^{-}=-1 , ignoring prompt-level success rates. However, under mini-batch stochastic optimization, 2-GRPO implicitly reweights prompts through their likelihood of forming contrastive pairs.
Standard GRPO relies on the empirical success rate p ^ q \hat{p}_{q} to estimate the true correctness probability p q p_{q} for advantage assignment, relying on larger group sizes for accuracy. While this mechanism appears degenerate in 2-GRPO, we show that, through the lens of stochastic optimization, 2-GRPO implicitly estimates the advantage.
Moreover, 2-GRPO does not simply estimate the large-group GRPO gradient with fewer samples; it induces a different prompt-level weighting that prioritizes prompts likely to yield contrastive pairs.
###### Proposition C.1 .
Given a constant p β ( 0 , 1 ) p\in(0,1) and a small positive constant Ο΅ \epsilon , we consider two scenarios below: β’ Case 1 : Consider X 1 , β― , X 2 β N βΌ i.i.d. Bernoulli β ( p ) X_{1},\cdots,X_{2N}\stackrel{{\scriptstyle\text{i.i.d.}}}{{\sim}}\text{Bernoulli}(p) . Let Y i = X i β ΞΌ ^ Ο ^ + Ο΅ Y_{i}=\frac{X_{i}-\hat{\mu}}{\hat{\sigma}+\epsilon} , where ΞΌ ^ = 1 2 β N β β i = 1 2 β N X i \hat{\mu}=\frac{1}{2N}\sum_{i=1}^{2N}X_{i} and Ο ^ = 1 2 β N β β i = 1 2 β N ( X i β ΞΌ ^ ) 2 \hat{\sigma}=\sqrt{\frac{1}{2N}\sum_{i=1}^{2N}\left(X_{i}-\hat{\mu}\right)^{2}} . Then, it follows that lim Ο΅ β 0 lim N β β πΌ β‘ [ Y i | X i = x ] = x β p p β‘ ( 1 β p ) . \lim_{\epsilon\rightarrow 0}\lim_{N\rightarrow\infty}{\mathbb{E}}[Y_{i}|X_{i}=x]=\frac{x-p}{\sqrt{p(1-p)}}. (26)
β’ Case 2 : Consider N N pairs of ( X i , 1 , X i , 2 ) (X_{i,1},X_{i,2}) with each X i , j βΌ i.i.d. Bernoulli β ( p ) X_{i,j}\stackrel{{\scriptstyle\text{i.i.d.}}}{{\sim}}\text{Bernoulli}(p) . Let Y i , j = X i , j β ΞΌ ^ i Ο ^ i + Ο΅ Y_{i,j}=\frac{X_{i,j}-\hat{\mu}_{i}}{\hat{\sigma}_{i}+\epsilon} , where ΞΌ ^ i = 1 2 β ( X i , 1 + X i , 2 ) \hat{\mu}_{i}=\frac{1}{2}(X_{i,1}+X_{i,2}) and Ο ^ i = 1 2 β β j = 1 2 ( X i , j β ΞΌ ^ i ) 2 \hat{\sigma}_{i}=\sqrt{\frac{1}{2}\sum_{j=1}^{2}(X_{i,j}-\hat{\mu}_{i})^{2}} . Then, it follows that lim Ο΅ β 0 lim N β β πΌ β‘ [ Y i , j | X i , j = x ] = x β p . \lim_{\epsilon\rightarrow 0}\lim_{N\to\infty}{\mathbb{E}}[Y_{i,j}|X_{i,j}=x]=x-p. (27)
Term lim Ο΅ β 0 , N β β πΌ β‘ [ Y i , j | X i , j = x ] \lim_{\epsilon\rightarrow 0,N\to\infty}{\mathbb{E}}[Y_{i,j}|X_{i,j}=x] differs from lim Ο΅ β 0 , N β β πΌ β‘ [ Y i | X i = x ] \lim_{\epsilon\rightarrow 0,N\rightarrow\infty}{\mathbb{E}}[Y_{i}|X_{i}=x] by a scaling factor 1 p β‘ ( 1 β p ) \frac{1}{\sqrt{p(1-p)}} .
In Proposition C.1 (proof in Appx. B.9 ), Case 1 corresponds to regular GRPO with sufficiently large group size. In this case, πΌ β‘ [ Y i | X i = 1 ] {\mathbb{E}}[Y_{i}|X_{i}=1] and πΌ β‘ [ Y i | X i = 0 ] {\mathbb{E}}[Y_{i}|X_{i}=0] are, respectively, the advantage estimates of positive and negative trajectories given a prompt, dependent on the success probability p q p_{q} . A large G G will lead to a better estimate of the success probability p q p_{q} . Case 2 corresponds to 2-GRPO, where πΌ β‘ [ Y i , j | X i , j = 1 ] {\mathbb{E}}[Y_{i,j}|X_{i,j}=1] and πΌ β‘ [ Y i , j | X i , j = 0 ] {\mathbb{E}}[Y_{i,j}|X_{i,j}=0] are advantage estimates, which are also dependent on the success rate p q p_{q} , amortizing over multiple stochastic updates.
2-GRPO produces advantage estimates that differ from standard GRPO solely by a scaling factor; this factor is effectively a design choice. Whether such a scaling is beneficial remains an open question [ 19 ] .
### C.2 Key of Variance Reduction: the Training Batch Size, not the Group Size
Beyond the inherent variance reduction mechanisms of PPO and GRPO, it is generally understood that using a larger group of rollouts yields a lower-variance policy gradient estimate. However, this perspective overlooks the practicalities of mini-batch optimization. In this section, we analyze the practical gradient variance within a mini-batch setting. To facilitate this discussion, we focus strictly on the optimization phase and treat the sampled rollouts as fixed training data for notational simplicity.
Note that there are two notions of βbatch sizeβ in VERL : data.train_batch_size denotes the rollout-generation batch size (by # prompts), whereas actor.ppo_mini_batch_size denotes the optimization mini-batch size (by # prompts). However, the effective number of samples during optimization is actually actor.ppo_mini_batch_size * rollout.n , counted by the number of rollouts.
###### Definition C.2 (Variance of Gradient Estimate in Mini-Batch) .
Without loss of generality, let { π i } i = 1 B \{{\bm{x}}_{i}\}_{i=1}^{B} be a batch of B B random variables (r.v.βs), where each π i {\bm{x}}_{i} is i.i.d. π βΌ π {\bm{x}}\sim{\mathcal{D}} , and let π β‘ ( π i ) = β ΞΈ L ΞΈ β ( π i ) \bm{g}({\bm{x}}_{i})=\nabla_{\theta}L_{\theta}({\bm{x}}_{i}) denote the gradient of L ΞΈ β ( π i ) L_{\theta}({\bm{x}}_{i}) w.r.t. ΞΈ \theta . Define the empirical batch gradient π ^ B = 1 B β β i = 1 B π β‘ ( π i ) \hat{\bm{g}}_{B}=\frac{1}{B}\sum_{i=1}^{B}\bm{g}({\bm{x}}_{i}) . Note that π β‘ ( π i ) \bm{g}({\bm{x}}_{i}) and π ^ B \hat{\bm{g}}_{B} are dependent r.v.βs of π i \bm{x}_{i} and { π i } i = 1 B \{\bm{x}_{i}\}_{i=1}^{B} , respectively. We denote the expectation of the gradient π Β― = πΌ π βΌ π β [ π β ( π ) ] \bar{\bm{g}}=\mathbb{E}_{{\bm{x}}\sim{\mathcal{D}}}[\bm{g}({\bm{x}})] . The variance of the gradient estimate over the batch is then defined as: Var β‘ ( π ^ B ) = Var { π i } i B β ( π ^ B ) = πΌ { π i } i B β ( ( π ^ B β π Β― ) 2 ) . {\mathrm{Var}}(\hat{\bm{g}}_{B})={\mathrm{Var}}_{\{{\bm{x}}_{i}\}_{i}^{B}}(\hat{\bm{g}}_{B})={\mathbb{E}}_{\{{\bm{x}}_{i}\}_{i}^{B}}\Big((\hat{\bm{g}}_{B}-\bar{\bm{g}})^{2}\Big). (28)
Following the definition of Variance of Gradient Estimate in Mini-batch (Def. C.2 ), we provide a lemma for its relationship to the mini-batch size.
###### Lemma C.3 .
Let { π± i } i = 1 B 1 , { π± i } i = 1 B 2 \{{\bm{x}}_{i}\}_{i=1}^{B_{1}},\{{\bm{x}}_{i}\}_{i=1}^{B_{2}} be two batches of B 1 B_{1} and B 2 B_{2} r.v.βs, respectively. Let g ^ B 1 , g ^ B 2 \hat{g}_{B_{1}},\hat{g}_{B_{2}} denote the empirical batch gradients of these two batches, respectively. If B 1 < B 2 B_{1}<B_{2} , then Var β‘ [ g ^ B 1 ] > Var β‘ [ g ^ B 2 ] \mathrm{Var}[\hat{g}_{B_{1}}]>\mathrm{Var}[\hat{g}_{B_{2}}] .
While decreasing the group size in Eq. ( 7 ) appears to increase the gradient variance for each individual prompt, this conclusion overlooks the total number of rollouts optimized across all prompts in a mini-batch, which is the effective number of examples for optimization. In Lemma C.3 (proof in Appx. B.10 ), we show that a larger batch size B B naturally leads to a lower variance of the gradient. Note that B B is the number of rollouts in each mini-batch rather than the number of prompts .
The actual calculation of GRPO is: π₯ ^ GRPO β ( ΞΈ , G , Q ) = 1 Q β G β β j = 1 Q β i = 1 G A i β j β Ο ΞΈ GRPO β ( o i β j | q j ) , \widehat{\mathcal{J}}_{\text{GRPO}}(\theta,G,Q)=\frac{1}{QG}\sum_{j=1}^{Q}\sum_{i=1}^{G}A_{ij}\pi_{\theta}^{\text{GRPO}}(o_{ij}|q_{j}), (29) where Ο ΞΈ GRPO β ( o | q ) = 1 G β β i = 1 G 1 | o i | β β t = 1 | o i | π Ο΅ β ( A i , t β Ο ΞΈ β ( o i , t | o i , < t , q ) Ο ΞΈ old β ( o i , t | o i , < t , q ) ) \pi_{\theta}^{\text{GRPO}}(o|q)=\frac{1}{G}\sum_{i=1}^{G}\frac{1}{|o_{i}|}\sum_{t=1}^{|o_{i}|}\mathcal{C}_{\epsilon}\left(A_{i,t}\frac{\pi_{\theta}(o_{i,t}|o_{i,<t},q)}{\pi_{\theta_{\text{old}}}(o_{i,t}|o_{i,<t},q)}\right) and Q Q is the number of prompts in the mini-batch, and the batch size w.r.t the number of rollouts is B = Q β G B=QG . When we decrease G G , we can increase Q Q to compensate to retain the same B B in a mini-batch. Since the total number of prompts in the dataset is fixed, increasing Q Q does not increase the total computational cost per training epoch.
### C.3 Exploration on Hard Questions
A difficult question often requires many attempts to yield a correct answer, which is necessary to form a valid contrastive signal. With a smaller group, the likelihood of sampling a correct response in a single iteration may appear lower, potentially raising concerns about degraded learning.
Under a fixed computational budget, 2-GRPO and 16-GRPO explore approximately the same total number of rollouts across all training epochs β the overall probability of sampling a correct answer under 2-GRPO is not lower than 16-GRPO, according to the Proposition C.4 .
###### Proposition C.4 .
Let p i β [ 0 , 1 ] p_{i}\in[0,1] denote the probability that a single rollout under the policy Ο i \pi_{i} produces a correct answer. Then: 1. The probability of obtaining at least one correct answer in 2 β m 2m independent rollouts with policy Ο 0 \pi_{0} is P 2 β m = 1 β ( 1 β p 0 ) 2 β m . P_{2m}=1-(1-p_{0})^{2m}. (30)