It Takes Two: Your GRPO Is Secretly DPO β€” part 2/5

Cleaned-text code points [15319, 30602). Bibliography omitted here; source.md includes it.

Based on previous analysis, the differences between GRPO and DPO are merely in the coefficients of contrastive gradient: GRPO: c ⁑ ( o i , t ∣ o i , < t , q ) := πŸ™ i , t Ο΅ ​ Var ^ ​ ( q ) | o i | ​ Ο€ ΞΈ old ​ ( o i , t ∣ o i , < t , q ) , \displaystyle c(o_{i,t}\mid o_{i,<t},q):=\frac{{\color[rgb]{0.043,0.2383,0.5703}\mathbbm{1}^{\epsilon}_{i,t}}\,{\color[rgb]{0.1055,0.3672,0.125}\sqrt{\widehat{\mathrm{Var}}(q)}}}{{\color[rgb]{0.7188,0.1094,0.1094}|o_{i}|}\,{\color[rgb]{0.043,0.2383,0.5703}\pi_{\theta_{\text{old}}}(o_{i,t}\mid o_{i,<t},q)}}\,, (8) DPO: c ⁑ ( o t ∣ o < t , q ) := Ξ² ​ Οƒ ​ ( r ^ ΞΈ ​ ( q , o βˆ’ ) βˆ’ r ^ ΞΈ ​ ( q , o + ) ) Ο€ ΞΈ ​ ( o ) , r ^ ΞΈ = Ξ² ​ log ⁑ Ο€ ΞΈ ​ ( y ∣ x ) Ο€ ref ​ ( y ∣ x ) . \displaystyle c(o_{t}\mid o_{<t},q):=\frac{\beta\,{\color[rgb]{0.1055,0.3672,0.125}\sigma(\hat{r}_{\theta}(q,o^{-})-\hat{r}_{\theta}(q,o^{+}))}}{{\color[rgb]{0.043,0.2383,0.5703}\pi_{\theta}(o)}}\,,\quad\hat{r}_{\theta}=\beta\log\frac{\pi_{\theta}(y\mid x)}{{\color[rgb]{0.2891,0.0781,0.5508}\pi_{\text{ref}}(y\mid x)}}\,. (9)

In the following, we show that the differences between GRPO and DPO are largely adaptations to their respective learning regimes: GRPO operates online with generated rollouts, whereas DPO operates offline with pre-collected preference data.

Group Size . DPO typically learns with fixed 1 1 -vs- 1 1 preference pairs which are collected offline in advance. By contrast, due to sampling responses online, GRPO needs to handle arbitrary N N -vs- M M positive–negative samples within each group. This changes only the Monte Carlo sample size used to estimate the same positive and negative contrastive gradients.

Token Aggregation . Within a sequence, GRPO averages token-level gradients, whereas DPO sums them. This is a design choice rather than a fundamental difference: e.g., SimPO Meng et al. (2024) – a DPO variant – uses mean aggregation, while Dr. GRPO Liu et al. (2025) – a GRPO variant – adopts sum aggregation.

Token-Level Weighting (Importance Sampling vs. Log-Likelihood). In GRPO, importance-sampling coefficients correct the gradient for samples generated by the old policy, specifically for its near-on-policy online RL setting. It is typically used together with clipping for training stability. DPO, however, does not require such correction in the offline setting and therefore directly uses the log-likelihood form of Ο€ ΞΈ \pi_{\theta} .

Group-Level Weighting . GRPO weights each group by Var ^ ​ ( q ) \sqrt{\widehat{\mathrm{Var}}(q)} , embodying its design philosophy of attending to more uncertain questions. DPO, in contrast, weights each pair by Οƒ ⁑ ( r ^ ΞΈ ​ ( q , o βˆ’ ) βˆ’ r ^ ΞΈ ​ ( q , o + ) ) \sigma(\hat{r}_{\theta}(q,o^{-})-\hat{r}_{\theta}(q,o^{+})) , assigning higher scores to pairs where the negative sample outscores the positive one.

Reference Model . DPO is regularized toward the reference model Ο€ ref \pi_{\text{ref}} through an implicit KL term. Optionally, GRPO can add a separate explicit KL penalty term w.r.t. the reference model.

In conclusion, the differences between GRPO and DPO mainly reflect adaptations to online vs. offline RL settings. The core mechanisms remain the same: both estimate a contrastive gradient that increases the likelihood of preferred outputs relative to unpreferred ones.

## 4 Why Viewing GRPO From Contrastive Learning?

### 4.1 Variance Reduction via Contrastive Objective

We demonstrate that this contrastive gradient formulation functions as a control variate method, where the coefficients serve to control the variance of the estimator.

###### Proposition 4.1 .

Let Ο€ ΞΈ \pi_{\theta} denote the policy model. Let o + ∼ Ο€ ΞΈ + ( β‹… | q ) o^{+}\sim\pi_{\theta}^{+}(\cdot|q) and o βˆ’ ∼ Ο€ ΞΈ βˆ’ ( β‹… | q ) o^{-}\sim\pi_{\theta}^{-}(\cdot|q) denote random variables representing a positive sample and a negative sample, respectively. Let 𝐠 + = βˆ‡ ΞΈ ​ log ​ Ο€ ΞΈ ​ ( o + | q ) \bm{g}^{+}=\nabla_{\theta}\log\pi_{\theta}(o^{+}|q) , 𝐠 βˆ’ = βˆ‡ ΞΈ ​ log ​ Ο€ ΞΈ ​ ( o βˆ’ | q ) \bm{g}^{-}=\nabla_{\theta}\log\pi_{\theta}(o^{-}|q) and ρ \rho denote the correlation coefficient of 𝐠 + \bm{g}^{+} and 𝐠 βˆ’ \bm{g}^{-} . If Cov ⁑ ( 𝐠 + , 𝐠 βˆ’ ) > 0 \mathrm{Cov}(\bm{g}^{+},\bm{g}^{-})>0 and 0 ≀ c ≀ 2 ​ Cov ⁑ ( 𝐠 + , 𝐠 βˆ’ ) Var ⁑ ( 𝐠 βˆ’ ) 0\leq c\leq 2\frac{\mathrm{Cov}(\bm{g}^{+},\bm{g}^{-})}{\mathrm{Var}(\bm{g}^{-})} , then Var ⁑ ( 𝐠 + βˆ’ c ​ 𝐠 βˆ’ ) ≀ Var ⁑ ( 𝐠 + ) \mathrm{Var}(\bm{g}^{+}-c\bm{g}^{-})\leq\mathrm{Var}(\bm{g}^{+}) . Specifically, if c = Cov ⁑ ( 𝐠 + , 𝐠 βˆ’ ) Var ⁑ ( 𝐠 βˆ’ ) c=\frac{\mathrm{Cov}(\bm{g}^{+},\bm{g}^{-})}{\mathrm{Var}(\bm{g}^{-})} , then Var ⁑ ( π’ˆ + βˆ’ c ​ π’ˆ βˆ’ ) = ( 1 βˆ’ ρ 2 ) ​ Var ​ ( π’ˆ + ) , \mathrm{Var}(\bm{g}^{+}-c\bm{g}^{-})=\left(1-\rho^{2}\right)\mathrm{Var}(\bm{g}^{+})\,, (10) where Var ​ ( β‹… ) \text{Var}(\cdot) and Cov ​ ( β‹… , β‹… ) \text{Cov}(\cdot,\cdot) denotes the corresponding traces of var/cov matrices for gradient vectors.

This proposition (proof in Appx. B.7 ) shows that, when the coefficient c c lies within an appropriate range, the variance of the gradient estimator can be reduced. This result directly follows the control variate method, a variance reduction technique widely used in Monte Carlo estimation and stochastic gradient optimization Johnson and Zhang (2013) . The reduction of gradient variance stabilizes RL training Li et al. (2024) .

A key implication of Proposition 4.1 is that the degree of variance reduction depends on the correlation between positive and negative samples. In LLM post-training, the positive sample o + o^{+} and the negative sample o βˆ’ o^{-} are generated by the same model conditioned on the same prompt q q , which typically induces a nontrivial correlation between them. (See Appx. B.8 for more discussion.)

### 4.2 GRPO with Small Group Size: It Should Fail, But Doesn’t

While both the value-baseline and contrastive perspectives account for GRPO’s variance reduction, they rest on fundamentally different assumptions. The prevailing value-baseline view holds that GRPO requires a sufficiently large group size to yield reliable group-level statistics; under this view, small-group GRPO should fail due to high-variance estimates Student (1908) . The contrastive perspective, by contrast, treats the positive and negative samples within a group as Monte Carlo estimates of the true positive and negative gradients. Because Monte Carlo estimation is unbiased regardless of sample size, GRPO with small groups should remain effective under stochastic optimization.

To adjudicate between these two views, we introduce 2-GRPO, a variant that uses the minimal group size of 2 2 . The value-baseline perspective predicts that this setting will fail (see Appx. B.2 for details). Empirically, however, 2-GRPO matches the performance of standard GRPO while achieving substantially higher efficiency, supporting the contrastive perspective as a more principled account of GRPO’s underlying mechanism. We describe 2-GRPO concretely in the following section.

### 4.3 Introducing 2-GRPO

With a group size of two, the GRPO advantage reduces to a simple contrastive signal: A + = + 1 A^{+}=+1 and A βˆ’ = βˆ’ 1 A^{-}=-1 when the two rollouts disagree on the reward, and both zeros otherwise. This yields an online RL counterpart of Direct Preference Optimization (DPO).

2-GRPO with Re-Sampling. Because of its binary contrastive nature, 2-GRPO discards any group whose two rollouts share the same reward. When the policy is highly accurate on the training set, this wastes a substantial fraction of generated samples and leads to sub-optimal performance. 3 3 3 A discussion on discard rate is provided in Appx. D.3 .

To address this, we introduce a resampling variant, denoted 2-GRPO+RS , which follows the strategy of DAPO Yu et al. (2025) : whenever a group is discarded due to a zero advantage, it is replaced with a fresh group sampled from a new prompt. Resampling adds rollout-stage computation, but the overhead is modest and consistently lifts peak performance.

Efficiency Gains. The efficiency gains of 2-GRPO arise at two stages: rollout generation and policy optimization. For a fixed number of prompts, 2-GRPO generates only 12.5 % 12.5\% of the rollouts required by 16-GRPO, and optimizes over the same 12.5 % 12.5\% fraction during policy updates. 2-GRPO+RS may generate additional rollouts through resampling, but its optimization-stage cost matches standard 2-GRPO. In our experiments, we cap the rollout budget of 2-GRPO+RS at that of 16-GRPO; in practice, it typically uses fewer.

## 5 Experiments

Goal of Experiment. Building on the theoretical justification for 2-GRPO, we seek to empirically assess its validity in RLVR. We anticipate that 2-GRPO will achieve a comparable performance as the regular GRPO (16-GRPO), and exhibit better efficiency β€”with respect to computational resources and/or wall-clock time.

Datasets, Baselines and Hyper-parameters. We provide the details of datasets, baselines and hyper-parameter choices in Appx. D.1 . For training, we adopt the verl framework Sheng et al. (2025) and utilize the built-in implementation of GRPO Shao et al. (2024) as the baseline algorithm.

### 5.1 Math Reasoning

Following prior studies Yu et al. (2025) , we consider mathematical tasks as representative instances of RLVR to verify our hypothesis. In the main experiment, the models are post-trained with RL techniques on MATH and DAPO-Math-Sub datasets under a fixed budget of 10 training epochs. The post-trained models are evaluated on five widely-used math reasoning benchmarks. This is an out-of-distribution evaluation setting, imposing requirements on the generalization ability of the post-trained models.

Table 1 showcases the Mean@32 as well as the training time. The empirical results show that 2-GRPO achieves 97.6% of 16-GRPO’s average performance while using only 12.5% of its total rollouts and 21.0% of its training time . 4 4 4 Appx. D.2 discusses the relationship between the total number of rollouts and computational cost.

These results provide strong corroboration of our theoretical finding that reducing group size preserves performance while substantially improving efficiency.

The resampling variant, 2-GRPO+RS, further improves peak performance, outperforming 16-GRPO on average while using roughly half of its training time . Although it is slower than 2-GRPO, it remains substantially more efficient than 16-GRPO, making it a practical alternative that preserves small-group efficiency while recovering the performance benefits of broader exploration.

Figure 2 shows the Pass@ K K over various K K choices. Overall, 2-GRPO achieves Pass@ K K performance comparable to 16-GRPO across different choices of K K . In particular, 2-GRPO even outperforms 16-GRPO on the AMC 2023 and Olympiad Bench. On AIME 2025, 2-GRPO performs better when post-trained on DAPO-Math-Sub, but worse when post-trained on MATH, likely due to the larger distribution shift between training dataset and the evaluation one.

We extend the evaluation of 2-GRPO to additional RLVR tasks beyond mathematical reasoning, including Vision Reasoning (Geometry3K) and Code Generation (Code-R1), with results reported in Figure 3 . The results demonstrate that 2-GRPO remains both effective and efficient across these diverse tasks, highlighting its broader applicability beyond math reasoning. As shown in the figure, 2-GRPO converges substantially faster than 16-GRPO, owing to the reduced number of samples generated and updated per step. This phenomenon is consistent with our theoretical findings, which identify the role of the group as providing contrastive sample pairs. Reducing the group size not only preserves performance but also accelerates the learning process.

### 5.2 Ablation Study: The Effect of Group Size

Our proposed 2-GRPO changes the group size while also adjusting the training batch size and the learning rate to account for the reduced number of rollouts per prompt (discussed in Appx. B.1 ). To isolate the effect of group size, we conduct an ablation study over different group sizes using the exact same configuration: 10 training epochs, a generation batch size of 512 prompts, a training batch size of 32 prompts, and a learning rate of 10 βˆ’ 6 10^{-6} .

It is worth noting that this setting is slightly unfavorable to smaller group sizes – the actual training mini-batch size by # rollouts is # prompts per batch multiplied by the group size. Therefore, GRPO with smaller group sizes in the ablation study suffered from higher variance of gradient estimates (see C.2 for details). Nonetheless, Figure 4 shows that the Mean@ 32 32 differences among G = 2 , 4 , 8 , 16 G=2,4,8,16 are consistently small across all settings. Moreover, increasing the group size does not reliably improve Pass@ 32 32 : larger groups do not consistently outperform smaller ones, and in some cases smaller groups achieve better Pass@ 32 32 . The full results of the ablation study are provided in Table 4 in Appx. D.4 .

## 6 Conclusion

In this work, we demonstrate that GRPO de facto functions as contrastive learning. We argue that the primary role of the group mechanism is not for accurate value-baseline estimation, as commonly assumed, but for the efficient construction of contrastive signals. Based on this insight, we reveal the fundamental connection between GRPO and DPOβ€”they are two echoes of the same contrastive gradient optimization principle, reflected through the online and offline RL settings, respectively. To further validate this insight, we introduce 2-GRPO, a minimal variant with only two rollouts per prompt. Although this setting is degenerate from the standpoint of traditional advantage estimation, it remains theoretically well motivated under our contrastive framework. Empirically, 2-GRPO achieves performance comparable to 16-GRPO while substantially reducing the computational overhead of rollout generation and policy optimization. These results support our hypothesis and suggest a more efficient design principle for RL algorithms for LLMs. More broadly, while our analysis focuses on GRPO, the insights developed here may extend to a wider class of group-based RL algorithms.

## Appendix

## Appendix A Related Work

### A.1 Contrastive Learning and LLM Alignment

Contrastive learning is the cornerstone of self-supervised representation learning [ 37 , 12 , 4 , 15 , 40 ] . The fundamental objective is to minimize the distance between anchor and positive samples in the representation space while maximizing the distance between the anchor and negative samples. Given this contrastive nature, the framework shares structural similarity with DPO, which conducts preference learning by increasing the likelihood of preferred completions relative to dispreferred ones. While recent literature explores the theoretical connections between RLHF and contrastive learning [ 13 , 8 , 24 ] , our work establishes a formal link between GRPO and DPO through a contrastive lens. This provides a unified analytical framework for understanding alignment. Specifically, we attribute the efficacy of GRPO to the construction of contrastive pairs, which serves as a control variate to reduce the variance of the gradient estimator. This analysis offers generalizable insights to broader alignment algorithms.

### A.2 Adaptive Rollouts in RLVR