Revisiting Reinforcement Learning with Verifiable Rewards from a Contrastive Perspective — part 1/4
Cleaned-text code points [0, 15923). Bibliography omitted here; source.md includes it.
# Revisiting Reinforcement Learning with Verifiable Rewards from a Contrastive Perspective
###### Abstract
Group Relative Policy Optimization (GRPO) is one of the most widely adopted RLVR algorithms for post-training large language models on reasoning tasks. We first show that GRPO admits an equivalent discriminative reformulation, in which policy optimization maximizes the expected score gap between verified positive and negative rollouts. This reformulation reveals two objective-level limitations: likelihood-misaligned surrogate scores , in which clipped ratio-based scores are optimized rather than the sequence likelihoods that govern generation, and score-insensitive credit assignment , in which rollout-level credit does not reflect the current score gaps between positive and negative rollouts. To address these limitations, we propose ConSPO, a Con trastive S equence-level P olicy O ptimization method that uses length-normalized sequence log-probabilities as rollout scores and contrasts verified positive rollouts against negative distractors within the same group. ConSPO optimizes a group-wise InfoNCE-style objective to adaptively strengthen updates for poorly separated positives and high-scoring negatives, together with a curriculum-scheduled margin that preserves separation pressure as training progresses. Experiments across diverse settings show that ConSPO outperforms strong baselines on challenging reasoning benchmarks.
## 1 Introduction
Reinforcement learning with verifiable rewards (RLVR) has become a standard paradigm for post-training LLMs on reasoning tasks. Among RLVR algorithms, Group Relative Policy Optimization (GRPO) ( 26 ; 11 ) is widely adopted for its critic-free design, which estimates group-relative advantages from outcome rewards within sampled rollout groups. Recent studies improve GRPO by modifying importance sampling to mitigate entropy collapse ( 39 ; 40 ; 9 ) , shaping rewards to control reasoning lengths ( 15 ; 34 ; 29 ) , or extending the framework to agentic applications ( 17 ; 5 ; 37 ) . Despite these advances, existing approaches mainly adjust GRPO around its original objective, leaving the core optimization mechanism underexplored.
Recent studies have highlighted the potential role of discriminative learning principles ( 27 ; 7 ) , providing a useful departure point from the standard group-relative advantage formulation. We make this connection explicit by rewriting GRPO as a weighted positive-negative discrimination objective (Proposition 1 ), whose general form and binary-reward specialization are summarized in Figure 1 . Under this view, GRPO increases the sequence-level scores of verified positive rollouts and decreases those of verified negative rollouts, where the scores are averages of clipped token-level importance sampling ratios. This reveals likelihood-misaligned surrogate scores : GRPO separates positive and negative rollouts using scores derived from clipped importance ratios, rather than the sequence likelihoods that determine generation. Consequently, GRPO can increase the surrogate score gap between positive and negative rollouts without improving their likelihood ordering under the current policy. This training-inference mismatch weakens the connection between objective improvement and generation quality, leading to suboptimal policy optimization ( 22 ) .
The binary-reward specialization further shows that GRPO reduces to positive-negative discrimination scaled by the group pass rate p ( q ) p(q) through the factor p ( q ) ( 1 − p ( q ) ) \sqrt{p(q)(1-p(q))} . This pass-rate weighting exposes score-insensitive credit assignment : GRPO determines rollout-level credit based on reward-based advantages tied to the group pass rate, rather than on current score gaps between positive and negative rollouts. As detailed by the gradient analysis in Lemma 1 and illustrated in Figure 2 , this mechanism assigns the same coefficient to all positive rollouts in a group and another coefficient to all negative rollouts. The resulting update is insensitive to within-group score gaps, treating a positive rollout close to negatives and a high-scoring hard negative equivalently to other rollouts with the same reward. Dynamic sampling mitigates this issue by retaining groups with intermediate pass rates ( 35 ; 1 ; 16 ) , but it does not remove the underlying source in the GRPO objective.
To address these limitations, we propose ConSPO, a Con trastive S equence-level P olicy O ptimization method for RLVR. ConSPO addresses likelihood-misaligned surrogate scores by replacing the clipped importance sampling ratio scores in GRPO with length-normalized sequence log-probabilities. This likelihood-based score directly matches the sequence probabilities that govern autoregressive generation, aligning training with inference. To address score-insensitive credit assignment , ConSPO optimizes a group-wise InfoNCE-style objective ( 24 ) , contrasting each verified positive rollout with negative distractors generated for the same question. As shown in Lemma 2 and Figure 2 , the softmax normalization makes the positive-rollout gradient depend on its separation from the negative distractors: the update is strengthened when the positive score remains close to the distractors and smoothly attenuated once it becomes well separated. For negative rollouts, the total suppressive signal increases when positives are poorly separated from the distractor set, with larger updates assigned to higher-scoring incorrect rollouts. ConSPO further introduces a margin to maintain separation pressure when positive rollouts only marginally outperform negative distractors. We schedule this margin as a curriculum ( 3 ; 30 ) , allowing optimization to begin with coarse positive-negative ordering and gradually shift toward stronger separation. Experiments across multiple backbone models, model scales, and training datasets show that ConSPO consistently improves performance on seven challenging mathematical reasoning benchmarks. Our main contributions are summarized as follows: • We derive an equivalent discriminative reformulation of GRPO that reveals two objective-level limitations: likelihood-misaligned surrogate scores and score-insensitive credit assignment .
• We propose ConSPO, which optimizes an InfoNCE-style sequence-level objective with likelihood-aligned scores and a curriculum-scheduled margin to address the above limitations.
• Experiments on challenging reasoning benchmarks show that ConSPO consistently outperforms strong RLVR baselines across different models, model scales, and training datasets.
## 2 Related Work
#### RLVR for LLM reasoning.
GRPO ( 11 ) estimates group-relative advantages from outcome rewards without a learned value model, motivating extensive efforts in RLVR for reasoning. Recent work improves GRPO by reallocating losses around the objective ( 20 ; 35 ) , adjusting importance sampling mechanisms for more stable optimization ( 39 ; 4 ; 9 ) , or shaping rewards toward more efficient reasoning ( 15 ; 34 ) . DisCO ( 18 ) reinterprets GRPO through a discriminative lens under binary rewards and explores alternative scoring functions, but does not address the score-insensitive credit assignment induced by the GRPO objective. ConSPO instead targets this limitation by optimizing an InfoNCE-style objective for online RLVR, with likelihood-aligned sequence scores and a curriculum-scheduled margin.
#### Contrastive objectives for policy alignment.
Contrastive learning distinguishes positive samples from negative distractors, commonly through InfoNCE-style objectives ( 24 ) , with representative methods enlarging negative sets via large-batch sampling or momentum queues ( 6 ; 13 ) . Recent policy alignment methods adopt similar discriminative principles to optimize policy likelihoods over preferred and dispreferred responses without explicit reward modeling ( 25 ; 31 ; 22 ) . Unlike these offline preference optimization methods, ConSPO transfers the contrastive principle to online RLVR, where positive-negative comparisons are constructed from verifiable rewards on policy-generated rollouts rather than from static pairwise preference data. Detailed related work is provided in Appendix B .
## 3 Preliminaries
Given a query q ∈ Q q\in Q , GRPO samples G G rollouts 𝐨 = { o i } i = 1 G \mathbf{o}=\{o_{i}\}_{i=1}^{G} from the old policy π old ( ⋅ ∣ q ) \pi_{\rm old}(\cdot\mid q) . Each rollout o i = ( o i , 1 , … , o i , | o i | ) o_{i}=(o_{i,1},\ldots,o_{i,|o_{i}|}) is generated autoregressively and receives a verifiable sequence-level reward r i = r ( o i , q ) r_{i}=r(o_{i},q) . The group-relative advantage is computed as A ^ i = ( r i − μ ^ q ) / σ ^ q \widehat{A}_{i}=(r_{i}-\widehat{\mu}_{q})/\widehat{\sigma}_{q} , where μ ^ q = 1 G ∑ j = 1 G r j , σ ^ q 2 = 1 G ∑ j = 1 G ( r j − μ ^ q ) 2 . \widehat{\mu}_{q}=\frac{1}{G}\sum_{j=1}^{G}r_{j},\qquad\widehat{\sigma}_{q}^{2}=\frac{1}{G}\sum_{j=1}^{G}(r_{j}-\widehat{\mu}_{q})^{2}.
For the analysis in Section 4 , we also use the corresponding expected advantage under π old ( ⋅ ∣ q ) \pi_{\rm old}(\cdot\mid q) . Let μ q \mu_{q} and σ q 2 \sigma_{q}^{2} denote the mean and variance of r ( o , q ) r(o,q) over o ∼ π old ( ⋅ ∣ q ) o\sim\pi_{\rm old}(\cdot\mid q) . Then A ( o ∣ q ) = r ( o , q ) − μ q σ q . A(o\mid q)=\frac{r(o,q)-\mu_{q}}{\sigma_{q}}. (1)
Subsequently, the policy parameters θ \theta are updated by maximizing a clipped surrogate objective, augmented with a KL divergence penalty to prevent severe deviation from a reference model π ref \pi_{\rm ref} :
𝒥 GRPO ( θ ) = 𝔼 q ∼ Q , { o i } i = 1 G ∼ π old ( ⋅ ∣ q ) \displaystyle\mathcal{J}_{\rm GRPO}(\theta)=\mathbb{E}_{q\sim Q,\,\{o_{i}\}_{i=1}^{G}\sim\pi_{\rm old}(\cdot\mid q)} (2) [ 1 G ∑ i = 1 G 1 | o i | ∑ t = 1 | o i | f ( ρ i , t ( θ ) , A ^ i ) − β D KL ( π θ ∥ π ref ) ] . \displaystyle\Bigg[\frac{1}{G}\sum_{i=1}^{G}\frac{1}{|o_{i}|}\sum_{t=1}^{|o_{i}|}f\!\left(\rho_{i,t}(\theta),\widehat{A}_{i}\right)-\beta D_{\rm KL}(\pi_{\theta}\|\pi_{\rm ref})\Bigg]. The token-level importance sampling ratio ρ i , t ( θ ) \rho_{i,t}(\theta) and clipping function f ( ρ , A ) f(\rho,A) are defined as ρ i , t ( θ ) \displaystyle\rho_{i,t}(\theta) = π θ ( o i , t ∣ q , o i , < t ) π old ( o i , t ∣ q , o i , < t ) , \displaystyle=\frac{\pi_{\theta}(o_{i,t}\mid q,o_{i,<t})}{\pi_{\rm old}(o_{i,t}\mid q,o_{i,<t})}, (3) f ( ρ , A ) \displaystyle f(\rho,A) = min ( ρ A , clip ( ρ , 1 − ϵ , 1 + ϵ ) A ) . \displaystyle=\min\!\left(\rho A,\,\operatorname{clip}(\rho,1-\epsilon,1+\epsilon)A\right).
## 4 Methods
In this section, we first reformulate GRPO as a weighted positive-negative discrimination objective, exposing its likelihood-misaligned surrogate scores and score-insensitive credit assignment. We then propose ConSPO, which optimizes a group-wise InfoNCE-style objective with likelihood-aligned sequence scores and incorporates a curriculum-scheduled margin to maintain separation pressure.
### 4.1 An Equivalent Discriminative Reformulation of GRPO
Let 𝒥 GRPO clip ( θ ) \mathcal{J}_{\rm GRPO}^{\rm clip}(\theta) denote the clipped policy surrogate in expectation form, obtained from Eq. ( 2 ) by removing the KL regularizer. Using the expected advantage A ( o ∣ q ) A(o\mid q) defined in Eq. ( 1 ), we show that this surrogate admits an equivalent discriminative reformulation.
###### Proposition 1 .
For queries with σ q > 0 \sigma_{q}>0 , the clipped GRPO surrogate 𝒥 GRPO clip ( θ ) \mathcal{J}_{\rm GRPO}^{\rm clip}(\theta) admits the following equivalent discriminative form: 𝒥 GRPO clip ( θ ) = 𝔼 q [ 1 2 𝔼 o ¯ ∼ π old ( ⋅ ∣ q ) [ | A ( o ¯ ∣ q ) | ] \displaystyle\mathcal{J}_{\rm GRPO}^{\rm clip}(\theta)=\mathbb{E}_{q}\Bigg[\frac{1}{2}\mathbb{E}_{\bar{o}\sim\pi_{\rm old}(\cdot\mid q)}\big[|A(\bar{o}\mid q)|\big] (4) 𝔼 o ∼ π ~ q + , o ′ ∼ π ~ q − [ s θ + ( o , q ) − s θ − ( o ′ , q ) ] ] . \displaystyle\mathbb{E}_{o\sim\widetilde{\pi}_{q}^{+},\,o^{\prime}\sim\widetilde{\pi}_{q}^{-}}\big[s_{\theta}^{+}(o,q)-s_{\theta}^{-}(o^{\prime},q)\big]\Bigg].
where π ~ q + \widetilde{\pi}_{q}^{+} and π ~ q − \widetilde{\pi}_{q}^{-} denote the normalized reweightings of π old ( ⋅ ∣ q ) \pi_{\rm old}(\cdot\mid q) by the positive and negative parts of A ( o ∣ q ) A(o\mid q) , respectively. In the common RLVR setting with binary rewards r ( o , q ) ∈ { 0 , 1 } r(o,q)\in\{0,1\} and valid queries satisfying 0 < p ( q ) < 1 0<p(q)<1 , Eq. ( 4 ) becomes 𝒥 GRPO clip ( θ ) = 𝔼 q p ( q ) ( 1 − p ( q ) ) \displaystyle\mathcal{J}_{\rm GRPO}^{\rm clip}(\theta)=\mathbb{E}_{q}\sqrt{p(q)(1-p(q))} (5) 𝔼 o ∼ π old + ( ⋅ ∣ q ) , o ′ ∼ π old − ( ⋅ ∣ q ) [ s θ + ( o , q ) − s θ − ( o ′ , q ) ] . \displaystyle\mathbb{E}_{o\sim\pi_{\rm old}^{+}(\cdot\mid q),\,o^{\prime}\sim\pi_{\rm old}^{-}(\cdot\mid q)}\big[s_{\theta}^{+}(o,q)-s_{\theta}^{-}(o^{\prime},q)\big]. where p ( q ) = 𝔼 o ∼ π old ( ⋅ ∣ q ) [ r ( o , q ) ] p(q)=\mathbb{E}_{o\sim\pi_{\rm old}(\cdot\mid q)}[r(o,q)] is the expected reward, i.e., the pass rate, under π old \pi_{\rm old} , and π old + ( ⋅ ∣ q ) \pi_{\rm old}^{+}(\cdot\mid q) and π old − ( ⋅ ∣ q ) \pi_{\rm old}^{-}(\cdot\mid q) denote π old \pi_{\rm old} conditioned on verified positive and negative outcomes, respectively.
The proof is provided in Appendix A.1 . The sequence-level surrogate scores s θ + s_{\theta}^{+} and s θ − s_{\theta}^{-} in Proposition 1 are obtained by averaging the clipped token-level importance sampling ratios:
s θ + ( o , q ) \displaystyle s_{\theta}^{+}(o,q) = 1 | o | ∑ t = 1 | o | min ( π θ ( o t ∣ q , o < t ) π old ( o t ∣ q , o < t ) , 1 + ϵ ) , \displaystyle=\frac{1}{|o|}\sum_{t=1}^{|o|}\min\left(\frac{\pi_{\theta}(o_{t}\mid q,o_{<t})}{\pi_{\rm old}(o_{t}\mid q,o_{<t})},1+\epsilon\right), (6) s θ − ( o ′ , q ) \displaystyle s_{\theta}^{-}(o^{\prime},q) = 1 | o ′ | ∑ t = 1 | o ′ | max ( π θ ( o t ′ ∣ q , o < t ′ ) π old ( o t ′ ∣ q , o < t ′ ) , 1 − ϵ ) . \displaystyle=\frac{1}{|o^{\prime}|}\sum_{t=1}^{|o^{\prime}|}\max\left(\frac{\pi_{\theta}(o^{\prime}_{t}\mid q,o^{\prime}_{<t})}{\pi_{\rm old}(o^{\prime}_{t}\mid q,o^{\prime}_{<t})},1-\epsilon\right). To make rollout-level credit assignment explicit, we instantiate the binary reformulation in Eq. ( 5 ) on a sampled group with N + > 0 N^{+}>0 and N − > 0 N^{-}>0 , and differentiate the resulting empirical objective with respect to the rollout scores.
###### Lemma 1 .
Consider a sampled valid group with a positive rollout set 𝒫 q = { o i + } i = 1 N + \mathcal{P}_{q}=\{o_{i}^{+}\}_{i=1}^{N^{+}} and a negative rollout set 𝒩 q = { o j − } j = 1 N − \mathcal{N}_{q}=\{o_{j}^{-}\}_{j=1}^{N^{-}} , and let p ^ q = N + / ( N + + N − ) \widehat{p}_{q}=N^{+}/(N^{+}+N^{-}) denote its empirical pass rate. Replacing the conditional expectations in Eq. ( 5 ) with empirical averages gives 𝒥 ^ GRPO clip ( q ) = p ^ q ( 1 − p ^ q ) \displaystyle\widehat{\mathcal{J}}_{\rm GRPO}^{\rm clip}(q)=\sqrt{\widehat{p}_{q}(1-\widehat{p}_{q})} (7) ( 1 N + ∑ i = 1 N + s θ + ( o i + , q ) − 1 N − ∑ j = 1 N − s θ − ( o j − , q ) ) . \displaystyle\left(\frac{1}{N^{+}}\sum_{i=1}^{N^{+}}s_{\theta}^{+}(o_{i}^{+},q)-\frac{1}{N^{-}}\sum_{j=1}^{N^{-}}s_{\theta}^{-}(o_{j}^{-},q)\right). The derivatives with respect to the positive and negative rollout scores are ∂ 𝒥 ^ GRPO clip ( q ) ∂ s θ + ( o i + , q ) \displaystyle\frac{\partial\widehat{\mathcal{J}}_{\rm GRPO}^{\rm clip}(q)}{\partial s_{\theta}^{+}(o_{i}^{+},q)} = 1 N + p ^ q ( 1 − p ^ q ) , \displaystyle=\frac{1}{N^{+}}\sqrt{\widehat{p}_{q}(1-\widehat{p}_{q})}, ∂ 𝒥 ^ GRPO clip ( q ) ∂ s θ − ( o j − , q ) \displaystyle\frac{\partial\widehat{\mathcal{J}}_{\rm GRPO}^{\rm clip}(q)}{\partial s_{\theta}^{-}(o_{j}^{-},q)} = − 1 N − p ^ q ( 1 − p ^ q ) . \displaystyle=-\frac{1}{N^{-}}\sqrt{\widehat{p}_{q}(1-\widehat{p}_{q})}.