It Takes Two: Your GRPO Is Secretly DPO — part 1/5

Cleaned-text code points [0, 15319). Bibliography omitted here; source.md includes it.

# It Takes Two: Your GRPO Is Secretly DPO

###### Abstract

GRPO has emerged as a prominent reinforcement learning algorithm for post-training LLMs. Unlike critic-based methods, GRPO computes advantages by estimating the value baselines from group-level statistics, eliminating the need for a critic network. Consequently, the prevailing view emphasizes the necessity of large group sizes, which are assumed to yield more accurate statistical estimates. In this paper, we propose a different view that the efficacy of GRPO stems from its implicit contrastive objective in the optimization, which helps reduce variance via the control variate method. This makes GRPO structurally related to preference learning methods such as DPO. This perspective motivates 2-GRPO, a minimal group-size variant that constructs contrastive signals with only two rollouts. We provide a rigorous theoretical analysis of 2-GRPO and empirically validate its effectiveness: 2-GRPO retains 97.6 % 97.6\% of the performance of 16-GRPO, while requiring only 12.5 % 12.5\% of the rollouts and 21 % 21\% of the training time.

## 1 Introduction

Reinforcement Learning (RL) has emerged as a central paradigm for the post-training of Large Language Models (LLMs). Two critical functions are aligning model outputs with human intent via RL with Human Feedback (RLHF) Ouyang et al. (2022) and enhancing reasoning capabilities through RL with Verifiable Rewards (RLVR) DeepSeek-AI (2025) . Among recent advances, Group Relative Policy Optimization (GRPO) Shao et al. (2024) is a prominent critic-free variant of Proximal Policy Optimization (PPO) Schulman et al. (2017) , which effectively reduces the variance of gradient estimates by subtracting the estimated value baseline. Diverging from PPO, which relies on an auxiliary critic network for estimating the value baselines, GRPO estimates the advantage function by sampling a group of responses (rollouts) for a single prompt and normalizing their rewards based on the group statistics (mean/standard deviation). This design eliminates the memory and computational overhead of the value network while maintaining strong performance across various reasoning tasks.

Conventional intuition suggests that GRPO’s efficacy is strongly correlated with its group size, grounded in the premise that larger sample sizes yield more accurate advantage estimates and lead to stronger post-trained LLMs. However, this intuition overlooks the specific construction of the group-relative gradient estimator in GRPO. First, we demonstrate that GRPO intrinsically functions as contrastive learning Chopra et al. (2005) and the contrastive objective effectively reduces the variance of the gradient estimates as a control variate method Johnson and Zhang (2013) . The group of rollouts serves primarily to pair contrastive samples, rather than to estimate the value baselines. Specifically, the GRPO objective is de facto a Monte Carlo estimator to approximate the true contrastive gradients. Thus, the choices of group size primarily affects the variance of the Monte Carlo estimator, while the approximation itself remains unbiased. Therefore, in contrast to the prevailing value-baseline estimation viewpoint, GRPO with a small group size shall still work properly. Second, this perspective further reveals its close connection to the well-known Direct Preference Optimization (DPO) algorithm Rafailov et al. (2023) , which explicitly introduces the contrastive objective in the offline RLHF setting. The GRPO is de facto doing direct preference optimization on online RL settings with the necessary adaptations.

To evidence this hypothesis, we propose the minimal two-rollout setting (2-GRPO), a configuration previously regarded as inadequate for estimating group statistics Student (1908) , but well aligned with the contrastive learning interpretation and the DPO objectives. We provide a thorough theoretical analysis of the properties of 2-GRPO and empirically evaluate its effectiveness and efficiency across a diverse set of models and tasks. The theoretical analysis justifies the rationale behind the 2-GRPO designs. Empirically, 2-GRPO achieves performance comparable to 16-GRPO while substantially reducing training time. We further propose a resampling variant, 2-GRPO+RS, which reduces sample discard rate and achieves performance closer to 16-GRPO while being more efficient than 16-GRPO. These findings support our central hypothesis: GRPO derives its strength primarily from its contrastive formulation, rather than from accurate advantage estimation. The efficiency of 2-GRPO further highlights the promise of the contrastive policy optimization direction.

## 2 Preliminary

### 2.1 Problem Setting and Notation

Our work focuses on RL-based post-training of LLMs for reasoning capabilities. Given an input prompt q ∈ 𝒬 q\in\mathcal{Q} , the model generates the i i -th response o i = ( o i , 1 , … , o i , T ) o_{i}=(o_{i,1},\ldots,o_{i,T}) , where o i , t o_{i,t} is the token generated at step t ∈ [ 0 , T ] t\in[0,T] and o i , < t o_{i,<t} denotes the sequence of preceding tokens. A trajectory τ = ( q , o ) ∈ 𝒯 \tau=(q,o)\in\mathcal{T} is defined as a concatenation of a prompt and its corresponding generated response. In current RL post-training, the reward function r : 𝒯 → ℝ r:\mathcal{T}\rightarrow\mathbb{R} is typically defined at the trajectory level. The learning objective is to maximize the expected reward over the trajectory space: 𝒥 ( θ ) = 𝔼 q ∼ 𝒬 𝔼 o ∼ π θ ( ⋅ | q ) [ r ( τ ) ] , \mathcal{J}(\theta)={\mathbb{E}}_{q\sim\mathcal{Q}}{\mathbb{E}}_{o\sim\pi_{\theta}(\cdot|q)}[r(\tau)]\,, (1) where π θ \pi_{\theta} denotes the policy model, a LLM with parameters θ \theta ; and 𝒬 \mathcal{Q} is the set of prompts, each consisting of a question and necessary instructions. We mainly focus on the setting of verifiable rewards, where the responses can be verified as correct ( r = 1 r=1 ) or incorrect ( r = 0 r=0 ).

### 2.2 The Story of Variance Reduction: VPG, PPO, and GRPO

As a foundational policy gradient method, Vanilla Policy Gradient (VPG) Williams (1992) optimizes the objective function using the following gradient estimator (where r i r_{i} is the reward of ( q , o i ) (q,o_{i}) ): ∇ θ 𝒥 ​ ( θ ) = 𝔼 q ∼ 𝒬 o i ∼ π θ ​ [ r i ​ ∑ t = 0 | o i | ∇ θ ​ log ​ π θ ​ ( o i , t | o i , < t , q ) ] . \nabla_{\theta}\mathcal{J}(\theta)=\underset{\begin{subarray}{c}q\sim\mathcal{Q}\\ o_{i}\sim\pi_{\theta}\end{subarray}}{{\mathbb{E}}}\left[r_{i}\sum_{t=0}^{|o_{i}|}\nabla_{\theta}\log\pi_{\theta}(o_{i,t}|o_{i,<t},q)\right]\;. (2)

Although effective, VPG usually suffers from high variance of gradient estimates and training instability. Therefore, subsequent works Schulman et al. (2015) ; Schulman et al. (2017) utilize advantage estimates Baird (1993) to reduce the variance of the policy gradient estimator: A i , t = r i − b ⁡ ( q ) A_{i,t}=r_{i}-b(q) , where A i , t A_{i,t} is token-level advantage and b ⁡ ( q ) b(q) is the value baseline function (See Appx. B.1 for more details). An auxiliary LLM is employed as a critic to estimate this value baseline, such as in Proximal Policy Optimization (PPO) ( Schulman et al., 2017 ) : 𝒥 ⁡ ( θ ) = 𝔼 q ∼ 𝒬 o i ∼ π θ old ​ 1 | o i | ​ ∑ t = 1 | o i | min ⁡ { A i , t ​ ρ i , t , A i , t ​ 𝒞 ϵ ​ ( ρ i , t ) } , ρ i , t = π θ ​ ( o i , t ∣ o i , < t , q ) π θ old ​ ( o i , t ∣ o i , < t , q ) , \mathcal{J}(\theta)=\underset{\begin{subarray}{c}q\sim\mathcal{Q}\\ o_{i}\sim\pi_{\theta_{\text{old}}}\end{subarray}}{\mathbb{E}}\frac{1}{|o_{i}|}\sum_{t=1}^{|o_{i}|}\min\left\{A_{i,t}\rho_{i,t},A_{i,t}\mathcal{C}_{\epsilon}(\rho_{i,t})\right\}\,,\,\rho_{i,t}=\frac{\pi_{\theta}\!\left(o_{i,t}\mid o_{i,<t},q\right)}{\pi_{\theta_{\text{old}}}\!\left(o_{i,t}\mid o_{i,<t},q\right)}\,, (3) where π θ old \pi_{\theta_{\text{old}}} is the policy used to generate trajectories, while π θ \pi_{\theta} denotes the current policy being optimized. The ρ i , t \rho_{i,t} term is the introduced importance sampling technique for online (near-)on-policy RL while 𝒞 ϵ ​ ( x ) \mathcal{C}_{\epsilon}(x) denotes the clipping function within the interval [ 1 − ϵ , 1 + ϵ ] [1-\epsilon,1+\epsilon] .

To eliminate the substantial computational overhead and memory demands of the critic network, several studies Li et al. (2024) ; Ahmadian et al. (2024) ; Shao et al. (2024) propose estimating the baseline without a critic network. Specifically, GRPO estimates the advantage using the reward statistics from a group of generated responses: A i , t = r i − mean ​ ( 𝐫 ) std ​ ( 𝐫 ) + ϵ , A_{i,t}=\frac{r_{i}-\text{mean}(\mathbf{r})}{\text{std}(\mathbf{r})+\epsilon}\,, (4) where r i r_{i} is the reward for response o i o_{i} given query q q , and 𝐫 \mathbf{r} denotes the vector of rewards for G G sampled responses associated with q q . Therefore, it is generally believed that GRPO requires a sufficiently large group size to obtain accurate group-level statistics for advantage estimation. 1 1 1 Due to limited space, the comprehensive related work is provided in Appx. A .

## 3 A Tale of Two Algorithms: GRPO and DPO

At first glance, the objectives of GRPO and DPO appear distinct on different RL settings. We show that they are the twin objects of the contrastive RL objective under the online/offline RL setting, which can be seen from the gradient forms of GRPO and DPO. This finding provides a new theoretical analysis (Sec. 4 ) and motivates a more efficient yet effective algorithm (Sec. 4.2 ).

### 3.1 Contrastive Objective for Sequences

Contrastive Learning Chopra et al. (2005) has been a powerful learning paradigm in (self-)supervised learning, ranging from 1 1 -vs- 1 1 (one positive and one negative) objectives Rendle et al. (2009) to 1 1 -vs- N N Oord et al. (2018) and N N -vs- M M variants Frosst et al. (2019) . We first formalize the contrastive loss objective for sequences for further analysis.

###### Definition 3.1 (Contrastive Loss for Sequences) .

Let π θ \pi_{\theta} be a probabilistic model and 𝒟 \mathcal{D} be a data distribution. Consider an anchor sequence 𝐱 ∼ 𝒟 \mathbf{x}\sim\mathcal{D} , and let 𝒟 + ( ⋅ ∣ 𝐱 ) \mathcal{D}^{+}(\cdot\mid\mathbf{x}) and 𝒟 − ( ⋅ ∣ 𝐱 ) \mathcal{D}^{-}(\cdot\mid\mathbf{x}) denote the conditional distributions for positive and negative samples, respectively. Let y t y_{t} denote the t t -th token of sequence 𝒚 {\bm{y}} . A differentiable loss function ℒ \mathcal{L} is contrastive if its gradient holds the following form: ∇ θ ℒ \displaystyle\nabla_{\theta}\mathcal{L} = − 𝔼 𝒙 ∼ 𝒟 [ 𝔼 𝒚 + ∼ 𝒟 + ∑ t = 1 | 𝒚 + | c t + ∇ θ π θ ( 𝒚 + t | 𝒚 + < t , 𝒙 ) − 𝔼 𝒚 − ∼ 𝒟 − ∑ t = 1 | 𝒚 − | c t − ∇ θ π θ ( 𝒚 − t | 𝒚 − < t , 𝒙 ) ] , \displaystyle=-\underset{{\bm{x}}\sim\mathcal{D}}{{\mathbb{E}}}\Bigg[\underset{{\bm{y}}^{+}\sim\mathcal{D}^{+}}{{\mathbb{E}}}\sum_{t=1}^{|{\bm{y}}^{+}|}c_{t}^{+}\nabla_{\theta}\pi_{\theta}({\bm{y}}^{+}_{t}|{\bm{y}}^{+}_{<t},{\bm{x}})\quad-\underset{{\bm{y}}^{-}\sim\mathcal{D}^{-}}{{\mathbb{E}}}\sum_{t=1}^{|{\bm{y}}^{-}|}c_{t}^{-}\nabla_{\theta}\pi_{\theta}\left({\bm{y}}^{-}_{t}|{\bm{y}}^{-}_{<t},{\bm{x}}\right)\Bigg]\,, (5) where c t + c_{t}^{+} and c t − c_{t}^{-} are token-level coefficients depending on specific algorithm design.

We adopt token-level coefficients for generality, as sequence-level coefficients can be recovered as a special case. Furthermore, the number of positive ( N N ) and negative ( M M ) samples of each data point may vary depending on the specific designs, serving as a Monte Carlo estimator to approximate the true gradient in Eq. ( 5 ).

#### DPO is a 1-vs-1 contrastive learning

Direct Preference Optimization (DPO) Rafailov et al. (2023) is a dominant offline RLHF algorithms for LLMs: ℒ DPO = − 𝔼 ( q , o + , o − ) ∼ 𝒟 DPO ​ [ log ⁡ σ ⁡ ( β ​ log ⁡ π θ ​ ( o + | q ) π ref ​ ( o + | q ) − β ​ log ⁡ π θ ​ ( o − | q ) π ref ​ ( o − | q ) ) ] , \mathcal{L}_{\text{DPO}}=\underset{(q,o^{+},o^{-})\sim\mathcal{D}_{\text{DPO}}}{-\mathbb{E}}\left[\log\sigma\left(\beta\log\frac{\pi_{\theta}(o^{+}|q)}{\pi_{\text{ref}}(o^{+}|q)}-\beta\log\frac{\pi_{\theta}(o^{-}|q)}{\pi_{\text{ref}}(o^{-}|q)}\right)\right]\;, (6) where the preference pair ( q , o + , o − ) ∼ 𝒟 DPO (q,o^{+},o^{-})\sim\mathcal{D}_{\text{DPO}} are from precollected human-annotated data. σ \sigma denotes the sigmoid function. It is easy to show that DPO is a 1-vs-1 contrastive learning. We provide Lemma B.1 and its proof in Appx. B.5 for reference.

### 3.2 GRPO: N-vs-M Contrastive Learning

We demonstrate that GRPO effectively functions as a dynamic N N -vs- M M contrastive learning, where the group size G = N + M G=N+M is fixed, but the specific values of N N (positive samples) and M M (negative samples) are dynamic based on the sampled responses. Let G q + G^{+}_{q} and G q − G^{-}_{q} denote the counts of correct and incorrect trajectories, respectively. The GRPO objective function can be formulated as: 𝒥 GRPO ( θ , G ) = 𝔼 [ q ∼ 𝒬 ; { o j + , o k − } j , k G ∼ π θ old ( ⋅ | q ) ] \displaystyle\mathcal{J}_{\text{GRPO}}(\theta,G)={{\mathbb{E}}}_{\left[q\sim\mathcal{Q};\{o_{j}^{+},o_{k}^{-}\}_{j,k}^{G}\sim\pi_{\theta_{\text{old}}}(\cdot|q)\right]} (7) Var ^ G ​ ( q ) ​ [ 1 G q + ​ ∑ j = 1 G q + 1 | o j + | ​ ∑ t = 1 | o j + | 𝒞 ϵ + ​ ( ρ j , t ) ⏟ positive − 1 G q − ​ ∑ k = 1 G q − 1 | o k − | ​ ∑ t = 1 | o k − | 𝒞 ϵ − ​ ( ρ k , t ) ⏟ negative ] , \displaystyle{\displaystyle\sqrt{\widehat{\mathrm{Var}}_{G}(q)}}\Bigg[\underbrace{\frac{1}{G_{q}^{+}}\sum_{j=1}^{G_{q}^{+}}\frac{1}{|o_{j}^{+}|}\sum_{t=1}^{|o_{j}^{+}|}\mathcal{C}_{\epsilon}^{+}\left(\rho_{j,t}\right)}_{\text{positive}}-\underbrace{\frac{1}{G_{q}^{-}}\sum_{k=1}^{G_{q}^{-}}\frac{1}{|o_{k}^{-}|}\sum_{t=1}^{|o_{k}^{-}|}\mathcal{C}_{\epsilon}^{-}\left(\rho_{k,t}\right)}_{\text{negative}}\Bigg]\,, where o j + o_{j}^{+} and o k − o_{k}^{-} denote rollouts with correct and incorrect outcomes, respectively. Denoting p ^ θ old , q = G q + / G \hat{p}_{\theta_{\text{old}},q}=G^{+}_{q}/G , the term Var ^ G ​ ( q ) = ( 1 − p ^ θ old , q ) ​ p ^ θ old , q \widehat{\mathrm{Var}}_{G}(q)=(1-\hat{p}_{\theta_{\text{old}},q})\hat{p}_{\theta_{\text{old}},q} is the empirical variance of the G G sampled trajectories from the true Bernoulli ​ ( p old , q ) \text{Bernoulli}(p_{\text{old},q}) under the RLVR setting. 2 2 2 In subsequent parts, we omit the subscript θ old \theta_{\text{old}} of p p for brevity. For simplicity, we denote the upper and lower clippings as 𝒞 ϵ + ​ ( x ) = min ⁡ [ x , 1 + ϵ ] \mathcal{C}_{\epsilon}^{+}(x)=\min[x,1+\epsilon] and 𝒞 ϵ − ​ ( x ) = max ⁡ [ x , 1 − ϵ ] \mathcal{C}_{\epsilon}^{-}(x)=\max[x,1-\epsilon] , respectively.

The formulation in Eq. ( 7 ) provides the foundation for the following proposition, with a proof provided in Appx. B.3 . Despite the sophisticated algorithm design of GRPO, this proposition unveils its contrastive nature.

###### Proposition 3.2 .

The maximization of the GRPO objective is equivalent to the minimization of an N N -vs- M M contrastive loss estimator.

The derivation of Proposition 3.2 is based on the binary reward assumption to align with the RLVR setting. However, GRPO’s contrastive nature extends to continuous rewards inherently (as shown in Figure 1 ). Due to the space limit, we focus on the properties of GRPO on RLVR.

### 3.3 Echoes of Contrastiveness: GRPO and DPO