# Revisiting Reinforcement Learning with Verifiable Rewards from a Contrastive Perspective

###### Abstract

Group Relative Policy Optimization (GRPO) is one of the most widely adopted RLVR algorithms for post-training large language models on reasoning tasks. We first show that GRPO admits an equivalent discriminative reformulation, in which policy optimization maximizes the expected score gap between verified positive and negative rollouts. This reformulation reveals two objective-level limitations: likelihood-misaligned surrogate scores , in which clipped ratio-based scores are optimized rather than the sequence likelihoods that govern generation, and score-insensitive credit assignment , in which rollout-level credit does not reflect the current score gaps between positive and negative rollouts. To address these limitations, we propose ConSPO, a Con trastive S equence-level P olicy O ptimization method that uses length-normalized sequence log-probabilities as rollout scores and contrasts verified positive rollouts against negative distractors within the same group. ConSPO optimizes a group-wise InfoNCE-style objective to adaptively strengthen updates for poorly separated positives and high-scoring negatives, together with a curriculum-scheduled margin that preserves separation pressure as training progresses. Experiments across diverse settings show that ConSPO outperforms strong baselines on challenging reasoning benchmarks.

## 1 Introduction

Reinforcement learning with verifiable rewards (RLVR) has become a standard paradigm for post-training LLMs on reasoning tasks. Among RLVR algorithms, Group Relative Policy Optimization (GRPO) ( 26 ; 11 ) is widely adopted for its critic-free design, which estimates group-relative advantages from outcome rewards within sampled rollout groups. Recent studies improve GRPO by modifying importance sampling to mitigate entropy collapse ( 39 ; 40 ; 9 ) , shaping rewards to control reasoning lengths ( 15 ; 34 ; 29 ) , or extending the framework to agentic applications ( 17 ; 5 ; 37 ) . Despite these advances, existing approaches mainly adjust GRPO around its original objective, leaving the core optimization mechanism underexplored.

Recent studies have highlighted the potential role of discriminative learning principles ( 27 ; 7 ) , providing a useful departure point from the standard group-relative advantage formulation. We make this connection explicit by rewriting GRPO as a weighted positive-negative discrimination objective (Proposition 1 ), whose general form and binary-reward specialization are summarized in Figure 1 . Under this view, GRPO increases the sequence-level scores of verified positive rollouts and decreases those of verified negative rollouts, where the scores are averages of clipped token-level importance sampling ratios. This reveals likelihood-misaligned surrogate scores : GRPO separates positive and negative rollouts using scores derived from clipped importance ratios, rather than the sequence likelihoods that determine generation. Consequently, GRPO can increase the surrogate score gap between positive and negative rollouts without improving their likelihood ordering under the current policy. This training-inference mismatch weakens the connection between objective improvement and generation quality, leading to suboptimal policy optimization ( 22 ) .

The binary-reward specialization further shows that GRPO reduces to positive-negative discrimination scaled by the group pass rate p ⁡ ( q ) p(q) through the factor p ​ ( q ) ​ ( 1 − p ​ ( q ) ) \sqrt{p(q)(1-p(q))} . This pass-rate weighting exposes score-insensitive credit assignment : GRPO determines rollout-level credit based on reward-based advantages tied to the group pass rate, rather than on current score gaps between positive and negative rollouts. As detailed by the gradient analysis in Lemma 1 and illustrated in Figure 2 , this mechanism assigns the same coefficient to all positive rollouts in a group and another coefficient to all negative rollouts. The resulting update is insensitive to within-group score gaps, treating a positive rollout close to negatives and a high-scoring hard negative equivalently to other rollouts with the same reward. Dynamic sampling mitigates this issue by retaining groups with intermediate pass rates ( 35 ; 1 ; 16 ) , but it does not remove the underlying source in the GRPO objective.

To address these limitations, we propose ConSPO, a Con trastive S equence-level P olicy O ptimization method for RLVR. ConSPO addresses likelihood-misaligned surrogate scores by replacing the clipped importance sampling ratio scores in GRPO with length-normalized sequence log-probabilities. This likelihood-based score directly matches the sequence probabilities that govern autoregressive generation, aligning training with inference. To address score-insensitive credit assignment , ConSPO optimizes a group-wise InfoNCE-style objective ( 24 ) , contrasting each verified positive rollout with negative distractors generated for the same question. As shown in Lemma 2 and Figure 2 , the softmax normalization makes the positive-rollout gradient depend on its separation from the negative distractors: the update is strengthened when the positive score remains close to the distractors and smoothly attenuated once it becomes well separated. For negative rollouts, the total suppressive signal increases when positives are poorly separated from the distractor set, with larger updates assigned to higher-scoring incorrect rollouts. ConSPO further introduces a margin to maintain separation pressure when positive rollouts only marginally outperform negative distractors. We schedule this margin as a curriculum ( 3 ; 30 ) , allowing optimization to begin with coarse positive-negative ordering and gradually shift toward stronger separation. Experiments across multiple backbone models, model scales, and training datasets show that ConSPO consistently improves performance on seven challenging mathematical reasoning benchmarks. Our main contributions are summarized as follows: • We derive an equivalent discriminative reformulation of GRPO that reveals two objective-level limitations: likelihood-misaligned surrogate scores and score-insensitive credit assignment .

• We propose ConSPO, which optimizes an InfoNCE-style sequence-level objective with likelihood-aligned scores and a curriculum-scheduled margin to address the above limitations.

• Experiments on challenging reasoning benchmarks show that ConSPO consistently outperforms strong RLVR baselines across different models, model scales, and training datasets.

## 2 Related Work

#### RLVR for LLM reasoning.

GRPO ( 11 ) estimates group-relative advantages from outcome rewards without a learned value model, motivating extensive efforts in RLVR for reasoning. Recent work improves GRPO by reallocating losses around the objective ( 20 ; 35 ) , adjusting importance sampling mechanisms for more stable optimization ( 39 ; 4 ; 9 ) , or shaping rewards toward more efficient reasoning ( 15 ; 34 ) . DisCO ( 18 ) reinterprets GRPO through a discriminative lens under binary rewards and explores alternative scoring functions, but does not address the score-insensitive credit assignment induced by the GRPO objective. ConSPO instead targets this limitation by optimizing an InfoNCE-style objective for online RLVR, with likelihood-aligned sequence scores and a curriculum-scheduled margin.

#### Contrastive objectives for policy alignment.

Contrastive learning distinguishes positive samples from negative distractors, commonly through InfoNCE-style objectives ( 24 ) , with representative methods enlarging negative sets via large-batch sampling or momentum queues ( 6 ; 13 ) . Recent policy alignment methods adopt similar discriminative principles to optimize policy likelihoods over preferred and dispreferred responses without explicit reward modeling ( 25 ; 31 ; 22 ) . Unlike these offline preference optimization methods, ConSPO transfers the contrastive principle to online RLVR, where positive-negative comparisons are constructed from verifiable rewards on policy-generated rollouts rather than from static pairwise preference data. Detailed related work is provided in Appendix B .

## 3 Preliminaries

Given a query q ∈ Q q\in Q , GRPO samples G G rollouts 𝐨 = { o i } i = 1 G \mathbf{o}=\{o_{i}\}_{i=1}^{G} from the old policy π old ( ⋅ ∣ q ) \pi_{\rm old}(\cdot\mid q) . Each rollout o i = ( o i , 1 , … , o i , | o i | ) o_{i}=(o_{i,1},\ldots,o_{i,|o_{i}|}) is generated autoregressively and receives a verifiable sequence-level reward r i = r ⁡ ( o i , q ) r_{i}=r(o_{i},q) . The group-relative advantage is computed as A ^ i = ( r i − μ ^ q ) / σ ^ q \widehat{A}_{i}=(r_{i}-\widehat{\mu}_{q})/\widehat{\sigma}_{q} , where μ ^ q = 1 G ​ ∑ j = 1 G r j , σ ^ q 2 = 1 G ​ ∑ j = 1 G ( r j − μ ^ q ) 2 . \widehat{\mu}_{q}=\frac{1}{G}\sum_{j=1}^{G}r_{j},\qquad\widehat{\sigma}_{q}^{2}=\frac{1}{G}\sum_{j=1}^{G}(r_{j}-\widehat{\mu}_{q})^{2}.

For the analysis in Section 4 , we also use the corresponding expected advantage under π old ( ⋅ ∣ q ) \pi_{\rm old}(\cdot\mid q) . Let μ q \mu_{q} and σ q 2 \sigma_{q}^{2} denote the mean and variance of r ⁡ ( o , q ) r(o,q) over o ∼ π old ( ⋅ ∣ q ) o\sim\pi_{\rm old}(\cdot\mid q) . Then A ⁡ ( o ∣ q ) = r ⁡ ( o , q ) − μ q σ q . A(o\mid q)=\frac{r(o,q)-\mu_{q}}{\sigma_{q}}. (1)

Subsequently, the policy parameters θ \theta are updated by maximizing a clipped surrogate objective, augmented with a KL divergence penalty to prevent severe deviation from a reference model π ref \pi_{\rm ref} :

𝒥 GRPO ( θ ) = 𝔼 q ∼ Q , { o i } i = 1 G ∼ π old ( ⋅ ∣ q ) \displaystyle\mathcal{J}_{\rm GRPO}(\theta)=\mathbb{E}_{q\sim Q,\,\{o_{i}\}_{i=1}^{G}\sim\pi_{\rm old}(\cdot\mid q)} (2) [ 1 G ∑ i = 1 G 1 | o i | ∑ t = 1 | o i | f ( ρ i , t ( θ ) , A ^ i ) − β D KL ( π θ ∥ π ref ) ] . \displaystyle\Bigg[\frac{1}{G}\sum_{i=1}^{G}\frac{1}{|o_{i}|}\sum_{t=1}^{|o_{i}|}f\!\left(\rho_{i,t}(\theta),\widehat{A}_{i}\right)-\beta D_{\rm KL}(\pi_{\theta}\|\pi_{\rm ref})\Bigg]. The token-level importance sampling ratio ρ i , t ​ ( θ ) \rho_{i,t}(\theta) and clipping function f ⁡ ( ρ , A ) f(\rho,A) are defined as ρ i , t ​ ( θ ) \displaystyle\rho_{i,t}(\theta) = π θ ​ ( o i , t ∣ q , o i , < t ) π old ​ ( o i , t ∣ q , o i , < t ) , \displaystyle=\frac{\pi_{\theta}(o_{i,t}\mid q,o_{i,<t})}{\pi_{\rm old}(o_{i,t}\mid q,o_{i,<t})}, (3) f ⁡ ( ρ , A ) \displaystyle f(\rho,A) = min ⁡ ( ρ ​ A , clip ⁡ ( ρ , 1 − ϵ , 1 + ϵ ) ​ A ) . \displaystyle=\min\!\left(\rho A,\,\operatorname{clip}(\rho,1-\epsilon,1+\epsilon)A\right).

## 4 Methods

In this section, we first reformulate GRPO as a weighted positive-negative discrimination objective, exposing its likelihood-misaligned surrogate scores and score-insensitive credit assignment. We then propose ConSPO, which optimizes a group-wise InfoNCE-style objective with likelihood-aligned sequence scores and incorporates a curriculum-scheduled margin to maintain separation pressure.

### 4.1 An Equivalent Discriminative Reformulation of GRPO

Let 𝒥 GRPO clip ​ ( θ ) \mathcal{J}_{\rm GRPO}^{\rm clip}(\theta) denote the clipped policy surrogate in expectation form, obtained from Eq. ( 2 ) by removing the KL regularizer. Using the expected advantage A ⁡ ( o ∣ q ) A(o\mid q) defined in Eq. ( 1 ), we show that this surrogate admits an equivalent discriminative reformulation.

###### Proposition 1 .

For queries with σ q > 0 \sigma_{q}>0 , the clipped GRPO surrogate 𝒥 GRPO clip ​ ( θ ) \mathcal{J}_{\rm GRPO}^{\rm clip}(\theta) admits the following equivalent discriminative form: 𝒥 GRPO clip ( θ ) = 𝔼 q [ 1 2 𝔼 o ¯ ∼ π old ( ⋅ ∣ q ) [ | A ( o ¯ ∣ q ) | ] \displaystyle\mathcal{J}_{\rm GRPO}^{\rm clip}(\theta)=\mathbb{E}_{q}\Bigg[\frac{1}{2}\mathbb{E}_{\bar{o}\sim\pi_{\rm old}(\cdot\mid q)}\big[|A(\bar{o}\mid q)|\big] (4) 𝔼 o ∼ π ~ q + , o ′ ∼ π ~ q − [ s θ + ( o , q ) − s θ − ( o ′ , q ) ] ] . \displaystyle\mathbb{E}_{o\sim\widetilde{\pi}_{q}^{+},\,o^{\prime}\sim\widetilde{\pi}_{q}^{-}}\big[s_{\theta}^{+}(o,q)-s_{\theta}^{-}(o^{\prime},q)\big]\Bigg].

where π ~ q + \widetilde{\pi}_{q}^{+} and π ~ q − \widetilde{\pi}_{q}^{-} denote the normalized reweightings of π old ( ⋅ ∣ q ) \pi_{\rm old}(\cdot\mid q) by the positive and negative parts of A ⁡ ( o ∣ q ) A(o\mid q) , respectively. In the common RLVR setting with binary rewards r ⁡ ( o , q ) ∈ { 0 , 1 } r(o,q)\in\{0,1\} and valid queries satisfying 0 < p ⁡ ( q ) < 1 0<p(q)<1 , Eq. ( 4 ) becomes 𝒥 GRPO clip ​ ( θ ) = 𝔼 q ​ p ​ ( q ) ​ ( 1 − p ​ ( q ) ) \displaystyle\mathcal{J}_{\rm GRPO}^{\rm clip}(\theta)=\mathbb{E}_{q}\sqrt{p(q)(1-p(q))} (5) 𝔼 o ∼ π old + ( ⋅ ∣ q ) , o ′ ∼ π old − ( ⋅ ∣ q ) [ s θ + ( o , q ) − s θ − ( o ′ , q ) ] . \displaystyle\mathbb{E}_{o\sim\pi_{\rm old}^{+}(\cdot\mid q),\,o^{\prime}\sim\pi_{\rm old}^{-}(\cdot\mid q)}\big[s_{\theta}^{+}(o,q)-s_{\theta}^{-}(o^{\prime},q)\big]. where p ( q ) = 𝔼 o ∼ π old ( ⋅ ∣ q ) [ r ( o , q ) ] p(q)=\mathbb{E}_{o\sim\pi_{\rm old}(\cdot\mid q)}[r(o,q)] is the expected reward, i.e., the pass rate, under π old \pi_{\rm old} , and π old + ( ⋅ ∣ q ) \pi_{\rm old}^{+}(\cdot\mid q) and π old − ( ⋅ ∣ q ) \pi_{\rm old}^{-}(\cdot\mid q) denote π old \pi_{\rm old} conditioned on verified positive and negative outcomes, respectively.

The proof is provided in Appendix A.1 . The sequence-level surrogate scores s θ + s_{\theta}^{+} and s θ − s_{\theta}^{-} in Proposition 1 are obtained by averaging the clipped token-level importance sampling ratios:

s θ + ​ ( o , q ) \displaystyle s_{\theta}^{+}(o,q) = 1 | o | ​ ∑ t = 1 | o | min ⁡ ( π θ ​ ( o t ∣ q , o < t ) π old ​ ( o t ∣ q , o < t ) , 1 + ϵ ) , \displaystyle=\frac{1}{|o|}\sum_{t=1}^{|o|}\min\left(\frac{\pi_{\theta}(o_{t}\mid q,o_{<t})}{\pi_{\rm old}(o_{t}\mid q,o_{<t})},1+\epsilon\right), (6) s θ − ​ ( o ′ , q ) \displaystyle s_{\theta}^{-}(o^{\prime},q) = 1 | o ′ | ​ ∑ t = 1 | o ′ | max ⁡ ( π θ ​ ( o t ′ ∣ q , o < t ′ ) π old ​ ( o t ′ ∣ q , o < t ′ ) , 1 − ϵ ) . \displaystyle=\frac{1}{|o^{\prime}|}\sum_{t=1}^{|o^{\prime}|}\max\left(\frac{\pi_{\theta}(o^{\prime}_{t}\mid q,o^{\prime}_{<t})}{\pi_{\rm old}(o^{\prime}_{t}\mid q,o^{\prime}_{<t})},1-\epsilon\right). To make rollout-level credit assignment explicit, we instantiate the binary reformulation in Eq. ( 5 ) on a sampled group with N + > 0 N^{+}>0 and N − > 0 N^{-}>0 , and differentiate the resulting empirical objective with respect to the rollout scores.

###### Lemma 1 .

Consider a sampled valid group with a positive rollout set 𝒫 q = { o i + } i = 1 N + \mathcal{P}_{q}=\{o_{i}^{+}\}_{i=1}^{N^{+}} and a negative rollout set 𝒩 q = { o j − } j = 1 N − \mathcal{N}_{q}=\{o_{j}^{-}\}_{j=1}^{N^{-}} , and let p ^ q = N + / ( N + + N − ) \widehat{p}_{q}=N^{+}/(N^{+}+N^{-}) denote its empirical pass rate. Replacing the conditional expectations in Eq. ( 5 ) with empirical averages gives 𝒥 ^ GRPO clip ​ ( q ) = p ^ q ​ ( 1 − p ^ q ) \displaystyle\widehat{\mathcal{J}}_{\rm GRPO}^{\rm clip}(q)=\sqrt{\widehat{p}_{q}(1-\widehat{p}_{q})} (7) ( 1 N + ​ ∑ i = 1 N + s θ + ​ ( o i + , q ) − 1 N − ​ ∑ j = 1 N − s θ − ​ ( o j − , q ) ) . \displaystyle\left(\frac{1}{N^{+}}\sum_{i=1}^{N^{+}}s_{\theta}^{+}(o_{i}^{+},q)-\frac{1}{N^{-}}\sum_{j=1}^{N^{-}}s_{\theta}^{-}(o_{j}^{-},q)\right). The derivatives with respect to the positive and negative rollout scores are ∂ 𝒥 ^ GRPO clip ​ ( q ) ∂ s θ + ​ ( o i + , q ) \displaystyle\frac{\partial\widehat{\mathcal{J}}_{\rm GRPO}^{\rm clip}(q)}{\partial s_{\theta}^{+}(o_{i}^{+},q)} = 1 N + ​ p ^ q ​ ( 1 − p ^ q ) , \displaystyle=\frac{1}{N^{+}}\sqrt{\widehat{p}_{q}(1-\widehat{p}_{q})}, ∂ 𝒥 ^ GRPO clip ​ ( q ) ∂ s θ − ​ ( o j − , q ) \displaystyle\frac{\partial\widehat{\mathcal{J}}_{\rm GRPO}^{\rm clip}(q)}{\partial s_{\theta}^{-}(o_{j}^{-},q)} = − 1 N − ​ p ^ q ​ ( 1 − p ^ q ) . \displaystyle=-\frac{1}{N^{-}}\sqrt{\widehat{p}_{q}(1-\widehat{p}_{q})}.

The proof of Lemma 1 follows by direct differentiation and is provided in Appendix A.1 .

#### Implications.

The reformulation identifies two objective-level limitations of GRPO. First, Eq. ( 6 ) reveals likelihood-misaligned surrogate scores : the positive-negative separation in GRPO is measured by sequence-level averages of clipped importance ratios, rather than by the sequence likelihoods that govern generation. Consequently, enlarging the surrogate score gap in Eq. ( 5 ) does not necessarily improve the likelihood ordering between positive and negative rollouts under the current policy. Second, Lemma 1 reveals score-insensitive credit assignment : rollout-level credit is determined by coefficients tied to the empirical pass rate p ^ q ​ ( 1 − p ^ q ) \sqrt{\widehat{p}_{q}(1-\widehat{p}_{q})} , rather than by current positive-negative score gaps within the group. As a result, all positives share the same update coefficient and all negatives share another, making the objective insensitive to whether a positive remains poorly separated from negatives or whether a negative is a high-scoring hard distractor.

### 4.2 A Contrastive View of RLVR Optimization

Motivated by the limitations above, ConSPO uses likelihood-aligned sequence scores and group-wise positive-negative contrasts. Specifically, it scores each rollout by the length-normalized log-probability s θ ​ ( o , q ) = 1 | o | ​ ∑ t = 1 | o | log ⁡ π θ ​ ( o t ∣ q , o < t ) s_{\theta}(o,q)=\frac{1}{|o|}\sum_{t=1}^{|o|}\log\pi_{\theta}(o_{t}\mid q,o_{<t}) . Unlike the clipped surrogate scores s θ + s_{\theta}^{+} and s θ − s_{\theta}^{-} in Eq. ( 6 ), the same scoring function is applied to both positive and negative rollouts, placing all rollouts on the likelihood scale that governs autoregressive generation. Given the rollout sets 𝒫 q \mathcal{P}_{q} and 𝒩 q \mathcal{N}_{q} , ConSPO treats negative rollouts as distractors and optimizes the following InfoNCE-style objective:

𝒥 ^ NCE ​ ( q ) = 1 N + ​ ∑ i = 1 N + τ \displaystyle\widehat{\mathcal{J}}_{\rm NCE}(q)=\frac{1}{N^{+}}\sum_{i=1}^{N^{+}}\tau (8) log ⁡ exp ⁡ ( s θ ​ ( o i + , q ) / τ ) exp ⁡ ( s θ ​ ( o i + , q ) / τ ) + ∑ j = 1 N − exp ⁡ ( s θ ​ ( o j − , q ) / τ ) . \displaystyle\log\frac{\exp(s_{\theta}(o_{i}^{+},q)/\tau)}{\exp(s_{\theta}(o_{i}^{+},q)/\tau)+\sum_{j=1}^{N^{-}}\exp(s_{\theta}(o_{j}^{-},q)/\tau)}. where τ > 0 \tau>0 is the temperature parameter. Unlike the linear positive-negative score difference in Eq. ( 5 ), Eq. ( 8 ) compares each positive rollout against the negative set via softmax normalization. For each positive rollout o i + o_{i}^{+} , we define the contrastive probabilities assigned to itself and to its negative distractors as

P i + \displaystyle P_{i}^{+} = exp ⁡ ( s θ ​ ( o i + , q ) / τ ) exp ⁡ ( s θ ​ ( o i + , q ) / τ ) + ∑ k = 1 N − exp ⁡ ( s θ ​ ( o k − , q ) / τ ) , \displaystyle=\frac{\exp(s_{\theta}(o_{i}^{+},q)/\tau)}{\exp(s_{\theta}(o_{i}^{+},q)/\tau)+\sum_{k=1}^{N^{-}}\exp(s_{\theta}(o_{k}^{-},q)/\tau)}, (9) P i ​ j − \displaystyle P_{ij}^{-} = exp ⁡ ( s θ ​ ( o j − , q ) / τ ) exp ⁡ ( s θ ​ ( o i + , q ) / τ ) + ∑ k = 1 N − exp ⁡ ( s θ ​ ( o k − , q ) / τ ) . \displaystyle=\frac{\exp(s_{\theta}(o_{j}^{-},q)/\tau)}{\exp(s_{\theta}(o_{i}^{+},q)/\tau)+\sum_{k=1}^{N^{-}}\exp(s_{\theta}(o_{k}^{-},q)/\tau)}.

By construction, P i + + ∑ j = 1 N − P i ​ j − = 1 P_{i}^{+}+\sum_{j=1}^{N^{-}}P_{ij}^{-}=1 for each positive rollout o i + o_{i}^{+} . These probabilities make rollout-level credit depend on within-group scores, as formalized in Lemma 2 .

###### Lemma 2 .

For the InfoNCE-style objective in Eq. ( 8 ), the derivatives with respect to the positive and negative rollout scores are ∂ 𝒥 ^ NCE ​ ( q ) ∂ s θ ​ ( o i + , q ) \displaystyle\frac{\partial\widehat{\mathcal{J}}_{\rm NCE}(q)}{\partial s_{\theta}(o_{i}^{+},q)} = 1 N + ​ ( 1 − P i + ) , \displaystyle=\frac{1}{N^{+}}(1-P_{i}^{+}), (10) ∂ 𝒥 ^ NCE ​ ( q ) ∂ s θ ​ ( o j − , q ) \displaystyle\frac{\partial\widehat{\mathcal{J}}_{\rm NCE}(q)}{\partial s_{\theta}(o_{j}^{-},q)} = − 1 N + ∑ i = 1 N + P i ​ j − . \displaystyle=-\frac{1}{N^{+}}\sum_{i=1}^{N^{+}}P_{ij}^{-}. Moreover, the aggregate derivative over the negative rollout scores and the total derivative over the group satisfy ∑ j = 1 N − ∂ 𝒥 ^ NCE ​ ( q ) ∂ s θ ​ ( o j − , q ) = − 1 N + ∑ i = 1 N + ( 1 − P i + ) , \displaystyle\sum_{j=1}^{N^{-}}\frac{\partial\widehat{\mathcal{J}}_{\rm NCE}(q)}{\partial s_{\theta}(o_{j}^{-},q)}=-\frac{1}{N^{+}}\sum_{i=1}^{N^{+}}(1-P_{i}^{+}), (11) ∑ i = 1 N + ∂ 𝒥 ^ NCE ​ ( q ) ∂ s θ ​ ( o i + , q ) + ∑ j = 1 N − ∂ 𝒥 ^ NCE ​ ( q ) ∂ s θ ​ ( o j − , q ) = 0 . \displaystyle\sum_{i=1}^{N^{+}}\frac{\partial\widehat{\mathcal{J}}_{\rm NCE}(q)}{\partial s_{\theta}(o_{i}^{+},q)}+\sum_{j=1}^{N^{-}}\frac{\partial\widehat{\mathcal{J}}_{\rm NCE}(q)}{\partial s_{\theta}(o_{j}^{-},q)}=0. For any two negative rollouts o j − o_{j}^{-} and o k − o_{k}^{-} , the magnitudes of their negative-score derivatives satisfy | ∂ 𝒥 ^ NCE ​ ( q ) / ∂ s θ ​ ( o j − , q ) | | ∂ 𝒥 ^ NCE ​ ( q ) / ∂ s θ ​ ( o k − , q ) | = ∑ i = 1 N + P i ​ j − ∑ i = 1 N + P i ​ k − \displaystyle\frac{\left|\partial\widehat{\mathcal{J}}_{\rm NCE}(q)/\partial s_{\theta}(o_{j}^{-},q)\right|}{\left|\partial\widehat{\mathcal{J}}_{\rm NCE}(q)/\partial s_{\theta}(o_{k}^{-},q)\right|}=\frac{\sum_{i=1}^{N^{+}}P_{ij}^{-}}{\sum_{i=1}^{N^{+}}P_{ik}^{-}} (12) = exp ⁡ ( s θ ​ ( o j − , q ) − s θ ​ ( o k − , q ) τ ) . \displaystyle=\exp\left(\frac{s_{\theta}(o_{j}^{-},q)-s_{\theta}(o_{k}^{-},q)}{\tau}\right).

The proof is provided in Appendix A.2 . Lemma 2 shows that ConSPO induces contrast-sensitive credit assignment. For positive rollouts, the coefficient 1 − P i + 1-P_{i}^{+} increases when the positive rollout remains close to its negative distractors, and attenuates as it becomes well separated. For negative rollouts, Eq. ( 11 ) shows that the aggregate suppressive signal has magnitude 1 N + ​ ∑ i = 1 N + ( 1 − P i + ) \frac{1}{N^{+}}\sum_{i=1}^{N^{+}}(1-P_{i}^{+}) , which grows when positives receive low contrastive probabilities. This signal is then allocated across negatives according to P i ​ j − P_{ij}^{-} , so higher-scoring negatives receive larger derivative magnitudes than lower-scoring ones, as shown in Eq. ( 12 ). The balance relation in Eq. ( 11 ) further shows that positive and negative score derivatives cancel at the group level. Compared with Lemma 1 , ConSPO assigns rollout-level credit according to current within-group score comparisons, rather than fixed coefficients determined by group reward statistics.

### 4.3 Scheduled Margin as a Separation Curriculum

Although the InfoNCE-style objective in Eq. ( 8 ) enables contrast-sensitive credit assignment, its gradients may attenuate once positive rollouts only marginally outperform their negative distractors. This becomes more pronounced in later training steps, where positives and negatives may be correctly ordered but remain insufficiently separated. To preserve separation pressure, ConSPO introduces a scheduled margin that lowers the effective score of each positive rollout in the contrastive objective. The margin starts from zero, allowing early training to focus on coarse positive-negative ordering, and gradually increases to enforce stronger separation as optimization progresses. At training step t t , ConSPO maximizes the resulting margin-enhanced contrastive objective:

𝒥 ^ ConSPO ​ ( q ) = 1 N + ​ ∑ i = 1 N + τ \displaystyle\widehat{\mathcal{J}}_{\rm ConSPO}(q)=\frac{1}{N^{+}}\sum_{i=1}^{N^{+}}\tau (13) log ⁡ exp ⁡ ( ( s θ ​ ( o i + , q ) − m t ) / τ ) exp ⁡ ( ( s θ ​ ( o i + , q ) − m t ) / τ ) + ∑ j = 1 N − exp ⁡ ( s θ ​ ( o j − , q ) / τ ) . \displaystyle\log\frac{\exp((s_{\theta}(o_{i}^{+},q)-m_{t})/\tau)}{\exp((s_{\theta}(o_{i}^{+},q)-m_{t})/\tau)+\sum_{j=1}^{N^{-}}\exp(s_{\theta}(o_{j}^{-},q)/\tau)}. where m t ≥ 0 m_{t}\geq 0 is the scheduled margin. By subtracting m t m_{t} from the positive score, ConSPO requires a positive rollout to exceed its negative distractors by a larger margin before the contrastive gradient attenuates.

Let λ t ∈ [ 0 , 1 ] \lambda_{t}\in[0,1] denote the normalized training progress, and let α ∈ ( 0 , 1 ] \alpha\in(0,1] denote the fraction of training used for margin warmup. We define the effective schedule progress and the margin as ρ t = min ⁡ ( λ t α , 1 ) , m t = M 2 ​ ( 1 − cos ⁡ ( π ​ ρ t ) ) . \rho_{t}=\min\left(\frac{\lambda_{t}}{\alpha},1\right),\quad m_{t}=\frac{M}{2}\left(1-\cos(\pi\rho_{t})\right). where M M is the target margin. Thus, the margin smoothly increases from zero to M M during the first α \alpha portion of training and remains fixed afterward.

## 5 Experiments

### 5.1 Settings

#### Models.

We conduct experiments across diverse model families and scales to validate the generalization of ConSPO, including DeepSeek-R1-Distill-Qwen-1.5B ( 11 ) , DeepSeek-R1-Distill-Qwen-7B, DeepSeek-R1-Distill-Llama-8B, Qwen3-4B-Base ( 33 ) , and Llama-3.2-3B-Instruct ( 10 ) .

#### Training data.

We use the DeepScaleR-Preview-Dataset ( 28 ) which comprises approximately 40k mathematical problems drawn from AIME ( 39 ) , AMC ( 19 ) , Omni-MATH ( 8 ) , and Still ( 23 ) . We also use DAPO-Math-17k ( 35 ) to validate the effectiveness of ConSPO on different training data.

#### Benchmarks.

We evaluate our models on seven benchmarks, including AIME 2024, AIME 2025, AIME 2026, HMMT 2025 ( 2 ) , MATH500 ( 14 ) , AMC, and OlympiadBench ( 12 ) . We report pass@1 on most benchmarks, and avg@32 on AIME, HMMT, and AMC to mitigate the variance introduced by the relatively small size of these datasets ( 21 ) . More details are provided in Appendix C.1 .

#### Baselines.

We compare ConSPO against seven recent state-of-the-art RLVR methods, including (1) GRPO ; (2) DAPO , which refines the GRPO loss formulation and introduces dynamic sampling and overlong reward shaping; (3) Dr. GRPO , which removes the standard deviation in advantage estimation and reorganizes loss allocation; (4) DisCO , which revisits GRPO from a discriminative perspective and eliminates its difficulty bias; (5) GMPO , which introduces sequence-level geometric-mean importance sampling; (6) CISPO , which introduces soft clipping in importance sampling; and (7) SAPO , which replaces hard clipping with a smooth, temperature-controlled gate to adaptively control off-policy updates.

#### Implementation details.

During training, we set the temperature to 1.0 to ensure sufficient exploration and the learning rate to 2 × 10 − 6 2\times 10^{-6} . The maximum response length is 8K for DeepSeek-R1-Distill-Qwen and DeepSeek-R1-Distill-Llama, and 4K for all other models. All methods run for 4 epochs, except for Qwen3-4B-Base, which runs for 2 epochs due to its faster convergence. Following standard practice ( 11 ; 39 ) , we discard all-positive and all-negative groups during training, as they do not permit a valid positive-negative contrast. We set the contrastive temperature τ \tau to 10, the target margin M M to 0.01, and the margin warmup ratio α \alpha to 30%. Evaluations are conducted every 100 steps, with a temperature of 0.6, a top- p p of 0.95, and the same maximum response length used during training. The best performance for each method is reported. Experiments on the 1.5B, 3B, and 4B models are run on a single compute node, whereas those on the 7B and 8B models are distributed across four compute nodes. More details are provided in Appendix C.2 .

### 5.2 Main Results

#### Performance with reasoning models.

As shown in Tables 1 and 2 , ConSPO delivers consistent improvements on reasoning models. On DeepSeek-R1-Distill-Qwen-1.5B, ConSPO achieves the best average performance of 44.4, outperforming GRPO by 4.4 points and the strongest baseline by 1.6 points. Moreover, ConSPO achieves the highest performance across all seven benchmarks in this setting, indicating its generalization across different reasoning tasks. The same trend holds for larger reasoning models. On DeepSeek-R1-Distill-Qwen-7B and DeepSeek-R1-Distill-Llama-8B, ConSPO improves over the strongest baseline by 2.1 and 1.9 points, respectively. These results suggest that the proposed contrastive optimization scheme remains effective across different parameter scales.

#### Generalization across backbone models and training datasets.

As shown in Tables 3 and 5 , ConSPO generalizes well across both base and instruction-tuned models. On Qwen3-4B-Base, ConSPO achieves the best average performance of 34.0, outperforming the strongest baseline by 2.1 points. Notably, on the challenging HMMT25 benchmark, where the base model obtains only 0.9, ConSPO improves the score to 8.0, suggesting that our method can substantially enhance performance on difficult reasoning tasks where the initial model performs poorly. On Llama-3.2-3B-Instruct, whose base reasoning capability is substantially weaker, ConSPO still achieves the highest average performance of 29.2, improving over the strongest baseline by 0.6 points. Moreover, Table 4 shows that ConSPO remains effective when trained on the DAPO-Math-17k dataset, achieving an average performance of 44.1 and outperforming the strongest baseline by 1.9 points. These results demonstrate that ConSPO’s effectiveness is not tied to a specific backbone or training dataset.

### 5.3 Ablation Study

Table 6 ablates the key components of ConSPO on DeepSeek-R1-Distill-Qwen-1.5B. Replacing the InfoNCE-style objective with a linear positive-negative score difference causes the largest performance drop, reducing the average score from 44.4 to 42.7. This result shows that group-wise contrastive optimization is central to ConSPO, as it enables credit assignment to adapt to current within-group score comparisons. Replacing the length-normalized sequence log-probabilities with clipped importance sampling ratio scores also decreases the average score to 43.4, confirming the importance of aligning rollout scores with the likelihood scale that governs autoregressive generation. The scheduled margin further improves performance: removing the margin reduces the average score to 44.1, while using a fixed margin achieves 43.9. These results indicate that the contrastive objective, likelihood-aligned sequence scoring, and curriculum-scheduled margin play complementary roles in ConSPO’s effectiveness.

### 5.4 Parameter Study

Figure 3 studies the sensitivity of ConSPO to the contrastive temperature τ \tau , target margin M M , and margin warmup ratio α \alpha . When τ \tau is too small, the contrastive distribution becomes overly sharp, potentially concentrating the update on a few high-scoring negatives and leading to less stable optimization. For the margin schedule, performance drops when either the target margin or the warmup ratio is large, suggesting that ConSPO benefits from a moderate separation requirement introduced at a proper pace.

## 6 Conclusion

In this paper, we derive an equivalent discriminative reformulation of GRPO as a weighted positive-negative discrimination objective. This reformulation exposes two objective-level limitations: likelihood-misaligned surrogate scores , where GRPO optimizes clipped ratio-based scores rather than generation likelihoods, and score-insensitive credit assignment , where rollout-level credit is insensitive to positive-negative score gaps. To address these limitations, we propose ConSPO, which optimizes a group-wise InfoNCE-style objective with likelihood-aligned sequence scores and a curriculum-scheduled margin. Extensive experiments across diverse settings validate the effectiveness and generalization of ConSPO.

## Limitations

Despite our best efforts, this study has several limitations. First, although our experiments cover multiple backbone models, model scales, and training datasets, our empirical validation primarily focuses on models up to the 8B scale. While we additionally compare ConSPO and GRPO on a 32B model in Appendix D , more systematic evaluations at this scale and beyond remain an important direction for future work. Second, ConSPO is a preliminary exploration of RLVR from a contrastive optimization perspective. We instantiate this perspective with an InfoNCE-style group-wise objective, likelihood-aligned sequence scores, and a curriculum-scheduled margin. Broader choices of objective formulations and rollout scoring functions remain worth investigating to better understand and generalize contrast-sensitive credit assignment in RLVR.

## Ethical Considerations

This work studies reinforcement learning methods to improve LLM reasoning abilities and does not introduce new datasets that contain private, sensitive, or personally identifiable information. The potential negative impacts of ConSPO are similar to those associated with general LLM reasoning technologies, including risks from misuse or insufficiently supervised deployment. We emphasize the importance of careful evaluation and appropriate safeguards to ensure fair, safe, and responsible use of LLM systems.

## Acknowledgement

This work was supported by Qwen Business Unit through Alibaba Research Intern Program.

## References

Bae et al. (2026) S. Bae, J. Hong, M. Y. Lee, H. Kim, J. Nam, and D. Kwak Online difficulty filtering for reasoning oriented reinforcement learning . In Proceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers) , pp. 700–719 . Cited by: §1 .

Balunović et al. (2025) M. Balunović, J. Dekoninck, I. Petrov, N. Jovanović, and M. Vechev MathArena: evaluating llms on uncontaminated math competitions . SRI Lab, ETH Zurich . External Links: Link Cited by: §5.1 .

Bengio et al. (2009) Y. Bengio, J. Louradour, R. Collobert, and J. Weston Curriculum learning . In Proceedings of the 26th annual international conference on machine learning , pp. 41–48 . Cited by: §1 .

Chen et al. (2025) A. Chen, A. Li, B. Gong, B. Jiang, B. Fei, B. Yang, B. Shan, C. Yu, C. Wang, C. Zhu, C. Xiao, C. Du, C. Zhang, C. Qiao, C. Zhang, C. Du, C. Guo, D. Chen, D. Ding, D. Sun, D. Li, E. Jiao, H. Zhou, H. Zhang, H. Ding, H. Sun, H. Feng, H. Cai, H. Zhu, J. Sun, J. Zhuang, J. Cai, J. Song, J. Zhu, J. Li, J. Tian, J. Liu, J. Xu, J. Yan, J. Liu, J. He, K. Feng, K. Yang, K. Xiao, L. Han, L. Wang, L. Yu, L. Feng, L. Li, L. Zheng, L. Du, L. Yang, L. Zeng, M. Yu, M. Tao, M. Chi, M. Zhang, M. Lin, N. Hu, N. Di, P. Gao, P. Li, P. Zhao, Q. Ren, Q. Xu, Q. Li, Q. Wang, R. Tian, R. Leng, S. Chen, S. Chen, S. Shi, S. Weng, S. Guan, S. Yu, S. Li, S. Zhu, T. Li, T. Cai, T. Liang, W. Cheng, W. Kong, W. Li, X. Chen, X. Song, X. Luo, X. Su, X. Li, X. Han, X. Hou, X. Lu, X. Zou, X. Shen, Y. Gong, Y. Ma, Y. Wang, Y. Shi, Y. Zhong, Y. Duan, Y. Fu, Y. Hu, Y. Gao, Y. Fan, Y. Yang, Y. Li, Y. Hu, Y. Huang, Y. Li, Y. Xu, Y. Mao, Y. Shi, Y. Wenren, Z. Li, Z. Li, Z. Tian, Z. Zhu, Z. Fan, Z. Wu, Z. Xu, Z. Yu, Z. Lyu, Z. Jiang, Z. Gao, Z. Wu, Z. Song, and Z. Sun MiniMax-m1: scaling test-time compute efficiently with lightning attention . External Links: 2506.13585 , Link Cited by: Appendix B , §2 .

Chen et al. (2026) M. Chen, L. Sun, T. Li, sunhaoze, ZhouYijie, C. Zhu, H. Wang, J. Z. Pan, W. Zhang, H. Chen, F. Yang, Z. Zhou, and W. Chen ReSearch: learning to reason with search for LLMs via reinforcement learning . In The Thirty-ninth Annual Conference on Neural Information Processing Systems , External Links: Link Cited by: §1 .

Chen et al. (2020) T. Chen, S. Kornblith, M. Norouzi, and G. Hinton A simple framework for contrastive learning of visual representations . In International conference on machine learning , pp. 1597–1607 . Cited by: Appendix B , §2 .

Cui et al. (2026) S. Cui, P. Cheng, J. Song, Y. Gai, G. Zhang, Z. Yu, J. Lin, X. Jiang, and G. Jiang CLIPO: contrastive learning in policy optimization generalizes rlvr . arXiv preprint arXiv:2603.10101 . Cited by: §1 .

Gao et al. (2025a) B. Gao, F. Song, Z. Yang, Z. Cai, Y. Miao, Q. Dong, L. Li, C. Ma, L. Chen, R. Xu, Z. Tang, B. Wang, D. Zan, S. Quan, G. Zhang, L. Sha, Y. Zhang, X. Ren, T. Liu, and B. Chang Omni-MATH: a universal olympiad level mathematic benchmark for large language models . In The Thirteenth International Conference on Learning Representations , External Links: Link Cited by: §5.1 .

Gao et al. (2025b) C. Gao, C. Zheng, X. Chen, K. Dang, S. Liu, B. Yu, A. Yang, S. Bai, J. Zhou, and J. Lin Soft adaptive policy optimization . arXiv preprint arXiv:2511.20347 . Cited by: Appendix B , §1 , §2 .

Grattafiori et al. (2024) A. Grattafiori, A. Dubey, A. Jauhri, A. Pandey, A. Kadian, A. Al-Dahle, A. Letman, A. Mathur, A. Schelten, A. Vaughan, A. Yang, A. Fan, A. Goyal, A. Hartshorn, A. Yang, A. Mitra, A. Sravankumar, A. Korenev, A. Hinsvark, A. Rao, A. Zhang, A. Rodriguez, A. Gregerson, A. Spataru, B. Roziere, B. Biron, B. Tang, B. Chern, C. Caucheteux, C. Nayak, C. Bi, C. Marra, C. McConnell, C. Keller, C. Touret, C. Wu, C. Wong, C. C. Ferrer, C. Nikolaidis, D. Allonsius, D. Song, D. Pintz, D. Livshits, D. Wyatt, D. Esiobu, D. Choudhary, D. Mahajan, D. Garcia-Olano, D. Perino, D. Hupkes, E. Lakomkin, E. AlBadawy, E. Lobanova, E. Dinan, E. M. Smith, F. Radenovic, F. Guzmán, F. Zhang, G. Synnaeve, G. Lee, G. L. Anderson, G. Thattai, G. Nail, G. Mialon, G. Pang, G. Cucurell, H. Nguyen, H. Korevaar, H. Xu, H. Touvron, I. Zarov, I. A. Ibarra, I. Kloumann, I. Misra, I. Evtimov, J. Zhang, J. Copet, J. Lee, J. Geffert, J. Vranes, J. Park, J. Mahadeokar, J. Shah, J. van der Linde, J. Billock, J. Hong, J. Lee, J. Fu, J. Chi, J. Huang, J. Liu, J. Wang, J. Yu, J. Bitton, J. Spisak, J. Park, J. Rocca, J. Johnstun, J. Saxe, J. Jia, K. V. Alwala, K. Prasad, K. Upasani, K. Plawiak, K. Li, K. Heafield, K. Stone, K. El-Arini, K. Iyer, K. Malik, K. Chiu, K. Bhalla, K. Lakhotia, L. Rantala-Yeary, L. van der Maaten, L. Chen, L. Tan, L. Jenkins, L. Martin, L. Madaan, L. Malo, L. Blecher, L. Landzaat, L. de Oliveira, M. Muzzi, M. Pasupuleti, M. Singh, M. Paluri, M. Kardas, M. Tsimpoukelli, M. Oldham, M. Rita, M. Pavlova, M. Kambadur, M. Lewis, M. Si, M. K. Singh, M. Hassan, N. Goyal, N. Torabi, N. Bashlykov, N. Bogoychev, N. Chatterji, N. Zhang, O. Duchenne, O. Çelebi, P. Alrassy, P. Zhang, P. Li, P. Vasic, P. Weng, P. Bhargava, P. Dubal, P. Krishnan, P. S. Koura, P. Xu, Q. He, Q. Dong, R. Srinivasan, R. Ganapathy, R. Calderer, R. S. Cabral, R. Stojnic, R. Raileanu, R. Maheswari, R. Girdhar, R. Patel, R. Sauvestre, R. Polidoro, R. Sumbaly, R. Taylor, R. Silva, R. Hou, R. Wang, S. Hosseini, S. Chennabasappa, S. Singh, S. Bell, S. S. Kim, S. Edunov, S. Nie, S. Narang, S. Raparthy, S. Shen, S. Wan, S. Bhosale, S. Zhang, S. Vandenhende, S. Batra, S. Whitman, S. Sootla, S. Collot, S. Gururangan, S. Borodinsky, T. Herman, T. Fowler, T. Sheasha, T. Georgiou, T. Scialom, T. Speckbacher, T. Mihaylov, T. Xiao, U. Karn, V. Goswami, V. Gupta, V. Ramanathan, V. Kerkez, V. Gonguet, V. Do, V. Vogeti, V. Albiero, V. Petrovic, W. Chu, W. Xiong, W. Fu, W. Meers, X. Martinet, X. Wang, X. Wang, X. E. Tan, X. Xia, X. Xie, X. Jia, X. Wang, Y. Goldschlag, Y. Gaur, Y. Babaei, Y. Wen, Y. Song, Y. Zhang, Y. Li, Y. Mao, Z. D. Coudert, Z. Yan, Z. Chen, Z. Papakipos, A. Singh, A. Srivastava, A. Jain, A. Kelsey, A. Shajnfeld, A. Gangidi, A. Victoria, A. Goldstand, A. Menon, A. Sharma, A. Boesenberg, A. Baevski, A. Feinstein, A. Kallet, A. Sangani, A. Teo, A. Yunus, A. Lupu, A. Alvarado, A. Caples, A. Gu, A. Ho, A. Poulton, A. Ryan, A. Ramchandani, A. Dong, A. Franco, A. Goyal, A. Saraf, A. Chowdhury, A. Gabriel, A. Bharambe, A. Eisenman, A. Yazdan, B. James, B. Maurer, B. Leonhardi, B. Huang, B. Loyd, B. D. Paola, B. Paranjape, B. Liu, B. Wu, B. Ni, B. Hancock, B. Wasti, B. Spence, B. Stojkovic, B. Gamido, B. Montalvo, C. Parker, C. Burton, C. Mejia, C. Liu, C. Wang, C. Kim, C. Zhou, C. Hu, C. Chu, C. Cai, C. Tindal, C. Feichtenhofer, C. Gao, D. Civin, D. Beaty, D. Kreymer, D. Li, D. Adkins, D. Xu, D. Testuggine, D. David, D. Parikh, D. Liskovich, D. Foss, D. Wang, D. Le, D. Holland, E. Dowling, E. Jamil, E. Montgomery, E. Presani, E. Hahn, E. Wood, E. Le, E. Brinkman, E. Arcaute, E. Dunbar, E. Smothers, F. Sun, F. Kreuk, F. Tian, F. Kokkinos, F. Ozgenel, F. Caggioni, F. Kanayet, F. Seide, G. M. Florez, G. Schwarz, G. Badeer, G. Swee, G. Halpern, G. Herman, G. Sizov, Guangyi, Zhang, G. Lakshminarayanan, H. Inan, H. Shojanazeri, H. Zou, H. Wang, H. Zha, H. Habeeb, H. Rudolph, H. Suk, H. Aspegren, H. Goldman, H. Zhan, I. Damlaj, I. Molybog, I. Tufanov, I. Leontiadis, I. Veliche, I. Gat, J. Weissman, J. Geboski, J. Kohli, J. Lam, J. Asher, J. Gaya, J. Marcus, J. Tang, J. Chan, J. Zhen, J. Reizenstein, J. Teboul, J. Zhong, J. Jin, J. Yang, J. Cummings, J. Carvill, J. Shepard, J. McPhie, J. Torres, J. Ginsburg, J. Wang, K. Wu, K. H. U, K. Saxena, K. Khandelwal, K. Zand, K. Matosich, K. Veeraraghavan, K. Michelena, K. Li, K. Jagadeesh, K. Huang, K. Chawla, K. Huang, L. Chen, L. Garg, L. A, L. Silva, L. Bell, L. Zhang, L. Guo, L. Yu, L. Moshkovich, L. Wehrstedt, M. Khabsa, M. Avalani, M. Bhatt, M. Mankus, M. Hasson, M. Lennie, M. Reso, M. Groshev, M. Naumov, M. Lathi, M. Keneally, M. Liu, M. L. Seltzer, M. Valko, M. Restrepo, M. Patel, M. Vyatskov, M. Samvelyan, M. Clark, M. Macey, M. Wang, M. J. Hermoso, M. Metanat, M. Rastegari, M. Bansal, N. Santhanam, N. Parks, N. White, N. Bawa, N. Singhal, N. Egebo, N. Usunier, N. Mehta, N. P. Laptev, N. Dong, N. Cheng, O. Chernoguz, O. Hart, O. Salpekar, O. Kalinli, P. Kent, P. Parekh, P. Saab, P. Balaji, P. Rittner, P. Bontrager, P. Roux, P. Dollar, P. Zvyagina, P. Ratanchandani, P. Yuvraj, Q. Liang, R. Alao, R. Rodriguez, R. Ayub, R. Murthy, R. Nayani, R. Mitra, R. Parthasarathy, R. Li, R. Hogan, R. Battey, R. Wang, R. Howes, R. Rinott, S. Mehta, S. Siby, S. J. Bondu, S. Datta, S. Chugh, S. Hunt, S. Dhillon, S. Sidorov, S. Pan, S. Mahajan, S. Verma, S. Yamamoto, S. Ramaswamy, S. Lindsay, S. Lindsay, S. Feng, S. Lin, S. C. Zha, S. Patil, S. Shankar, S. Zhang, S. Zhang, S. Wang, S. Agarwal, S. Sajuyigbe, S. Chintala, S. Max, S. Chen, S. Kehoe, S. Satterfield, S. Govindaprasad, S. Gupta, S. Deng, S. Cho, S. Virk, S. Subramanian, S. Choudhury, S. Goldman, T. Remez, T. Glaser, T. Best, T. Koehler, T. Robinson, T. Li, T. Zhang, T. Matthews, T. Chou, T. Shaked, V. Vontimitta, V. Ajayi, V. Montanez, V. Mohan, V. S. Kumar, V. Mangla, V. Ionescu, V. Poenaru, V. T. Mihailescu, V. Ivanov, W. Li, W. Wang, W. Jiang, W. Bouaziz, W. Constable, X. Tang, X. Wu, X. Wang, X. Wu, X. Gao, Y. Kleinman, Y. Chen, Y. Hu, Y. Jia, Y. Qi, Y. Li, Y. Zhang, Y. Zhang, Y. Adi, Y. Nam, Yu, Wang, Y. Zhao, Y. Hao, Y. Qian, Y. Li, Y. He, Z. Rait, Z. DeVito, Z. Rosnbrick, Z. Wen, Z. Yang, Z. Zhao, and Z. Ma The llama 3 herd of models . External Links: 2407.21783 , Link Cited by: §5.1 .

Guo et al. (2025) D. Guo, D. Yang, H. Zhang, J. Song, P. Wang, Q. Zhu, R. Xu, R. Zhang, S. Ma, X. Bi, X. Zhang, X. Yu, Y. Wu, Z. F. Wu, Z. Gou, Z. Shao, Z. Li, Z. Gao, A. Liu, B. Xue, B. Wang, B. Wu, B. Feng, C. Lu, C. Zhao, C. Deng, C. Ruan, D. Dai, D. Chen, D. Ji, E. Li, F. Lin, F. Dai, F. Luo, G. Hao, G. Chen, G. Li, H. Zhang, H. Xu, H. Ding, H. Gao, H. Qu, H. Li, J. Guo, J. Li, J. Chen, J. Yuan, J. Tu, J. Qiu, J. Li, J. L. Cai, J. Ni, J. Liang, J. Chen, K. Dong, K. Hu, K. You, K. Gao, K. Guan, K. Huang, K. Yu, L. Wang, L. Zhang, L. Zhao, L. Wang, L. Zhang, L. Xu, L. Xia, M. Zhang, M. Zhang, M. Tang, M. Zhou, M. Li, M. Wang, M. Li, N. Tian, P. Huang, P. Zhang, Q. Wang, Q. Chen, Q. Du, R. Ge, R. Zhang, R. Pan, R. Wang, R. J. Chen, R. L. Jin, R. Chen, S. Lu, S. Zhou, S. Chen, S. Ye, S. Wang, S. Yu, S. Zhou, S. Pan, S. S. Li, S. Zhou, S. Wu, T. Yun, T. Pei, T. Sun, T. Wang, W. Zeng, W. Liu, W. Liang, W. Gao, W. Yu, W. Zhang, W. L. Xiao, W. An, X. Liu, X. Wang, X. Chen, X. Nie, X. Cheng, X. Liu, X. Xie, X. Liu, X. Yang, X. Li, X. Su, X. Lin, X. Q. Li, X. Jin, X. Shen, X. Chen, X. Sun, X. Wang, X. Song, X. Zhou, X. Wang, X. Shan, Y. K. Li, Y. Q. Wang, Y. X. Wei, Y. Zhang, Y. Xu, Y. Li, Y. Zhao, Y. Sun, Y. Wang, Y. Yu, Y. Zhang, Y. Shi, Y. Xiong, Y. He, Y. Piao, Y. Wang, Y. Tan, Y. Ma, Y. Liu, Y. Guo, Y. Ou, Y. Wang, Y. Gong, Y. Zou, Y. He, Y. Xiong, Y. Luo, Y. You, Y. Liu, Y. Zhou, Y. X. Zhu, Y. Huang, Y. Li, Y. Zheng, Y. Zhu, Y. Ma, Y. Tang, Y. Zha, Y. Yan, Z. Z. Ren, Z. Ren, Z. Sha, Z. Fu, Z. Xu, Z. Xie, Z. Zhang, Z. Hao, Z. Ma, Z. Yan, Z. Wu, Z. Gu, Z. Zhu, Z. Liu, Z. Li, Z. Xie, Z. Song, Z. Pan, Z. Huang, Z. Xu, Z. Zhang, and Z. Zhang DeepSeek-r1 incentivizes reasoning in llms through reinforcement learning . Nature 645 ( 8081 ), pp. 633–638 . External Links: ISSN 1476-4687 , Link , Document Cited by: Appendix B , §1 , §2 , §5.1 , §5.1 .

He et al. (2024) C. He, R. Luo, Y. Bai, S. Hu, Z. L. Thai, J. Shen, J. Hu, X. Han, Y. Huang, Y. Zhang, J. Liu, L. Qi, Z. Liu, and M. Sun OlympiadBench: a challenging benchmark for promoting agi with olympiad-level bilingual multimodal scientific problems . External Links: 2402.14008 , Link Cited by: §5.1 .

He et al. (2020) K. He, H. Fan, Y. Wu, S. Xie, and R. Girshick Momentum contrast for unsupervised visual representation learning . In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pp. 9729–9738 . Cited by: Appendix B , §2 .

Hendrycks et al. (2021) D. Hendrycks, C. Burns, S. Kadavath, A. Arora, S. Basart, E. Tang, D. Song, and J. Steinhardt Measuring mathematical problem solving with the math dataset . In Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2) , Cited by: §5.1 .

Huang et al. (2026) C. Huang, Z. Zhang, and C. Cardie Hapo: training language models to reason concisely via history-aware policy optimization . In Proceedings of the AAAI Conference on Artificial Intelligence , Vol. 40 , pp. 31122–31130 . Cited by: Appendix B , §1 , §2 .

Jiang et al. (2025) G. Jiang, W. Feng, G. Quan, C. Hao, Y. Zhang, G. Liu, and H. Wang Vcrl: variance-based curriculum reinforcement learning for large language models . arXiv preprint arXiv:2509.19803 . Cited by: §1 .

Jin et al. (2025) B. Jin, H. Zeng, Z. Yue, J. Yoon, S. O. Arik, D. Wang, H. Zamani, and J. Han Search-r1: training LLMs to reason and leverage search engines with reinforcement learning . In Second Conference on Language Modeling , External Links: Link Cited by: §1 .

Li et al. (2025) G. Li, M. Lin, T. Galanti, Z. Tu, and T. Yang DisCO: reinforcing large reasoning models with discriminative constrained optimization . In The Thirty-ninth Annual Conference on Neural Information Processing Systems , External Links: Link Cited by: Appendix B , §2 .

LI et al. (2024) J. LI, E. Beeching, L. Tunstall, B. Lipkin, R. Soletskyi, S. C. Huang, K. Rasul, L. Yu, A. Jiang, Z. Shen, Z. Qin, B. Dong, L. Zhou, Y. Fleureau, G. Lample, and S. Polu NuminaMath . Numina . Note: https://huggingface.co/datasets/AI-MO/NuminaMath-CoT Cited by: §5.1 .

Liu et al. (2025a) Z. Liu, C. Chen, W. Li, P. Qi, T. Pang, C. Du, W. S. Lee, and M. Lin Understanding r1-zero-like training: a critical perspective . In Second Conference on Language Modeling , External Links: Link Cited by: Appendix B , §2 .

Liu et al. (2025b) Z. Liu, C. Gong, X. Fu, Y. Liu, R. Chen, S. Hu, S. Zhang, R. Liu, Q. Zhang, and D. Tu Ghpo: adaptive guidance for stable and efficient llm reinforcement learning . arXiv preprint arXiv:2507.10628 . Cited by: §C.1 , §5.1 .

Meng et al. (2024) Y. Meng, M. Xia, and D. Chen Simpo: simple preference optimization with a reference-free reward . Advances in Neural Information Processing Systems 37 , pp. 124198–124235 . Cited by: Appendix B , §1 , §2 .

Min et al. (2024) Y. Min, Z. Chen, J. Jiang, J. Chen, J. Deng, Y. Hu, Y. Tang, J. Wang, X. Cheng, H. Song, W. X. Zhao, Z. Liu, Z. Wang, and J. Wen Imitate, explore, and self-improve: A reproduction report on slow-thinking reasoning systems . CoRR abs/2412.09413 . External Links: Link , Document , 2412.09413 Cited by: §5.1 .

Oord et al. (2018) A. v. d. Oord, Y. Li, and O. Vinyals Representation learning with contrastive predictive coding . arXiv preprint arXiv:1807.03748 . Cited by: Appendix B , §1 , §2 .

Rafailov et al. (2023) R. Rafailov, A. Sharma, E. Mitchell, C. D. Manning, S. Ermon, and C. Finn Direct preference optimization: your language model is secretly a reward model . Advances in neural information processing systems 36 , pp. 53728–53741 . Cited by: Appendix B , §2 .

Shao et al. (2024) Z. Shao, P. Wang, Q. Zhu, R. Xu, J. Song, X. Bi, H. Zhang, M. Zhang, Y. K. Li, Y. Wu, and D. Guo DeepSeekMath: pushing the limits of mathematical reasoning in open language models . External Links: 2402.03300 , Link Cited by: §1 .

Su et al. (2025) X. Su, S. Xie, G. Liu, Y. Xia, R. Luo, P. Jin, Z. Ma, Y. Wang, Z. Wang, and Y. Liu Trust region preference approximation: a simple and stable reinforcement learning algorithm for llm reasoning . arXiv preprint arXiv:2504.04524 . Cited by: §1 .

Tan et al. (2026) S. Tan, M. Luo, J. Wong, C. Cai, X. Shi, W. Y. Tang, M. Roongta, T. Zhang, L. E. Li, R. A. Popa, and I. Stoica DeepScaleR: effective RL scaling of reasoning models via iterative context lengthening . External Links: Link Cited by: §C.1 , §5.1 .

Tan et al. (2025) Z. Tan, H. Gao, X. Ma, F. Zhang, and Z. Dong Towards flash thinking via decoupled advantage policy optimization . arXiv preprint arXiv:2510.15374 . Cited by: §1 .

Wang et al. (2021) X. Wang, Y. Chen, and W. Zhu A survey on curriculum learning . IEEE transactions on pattern analysis and machine intelligence 44 ( 9 ), pp. 4555–4576 . Cited by: §1 .

Xu et al. (2024) H. Xu, A. Sharaf, Y. Chen, W. Tan, L. Shen, B. Van Durme, K. Murray, and Y. J. Kim Contrastive preference optimization: pushing the boundaries of llm performance in machine translation . arXiv preprint arXiv:2401.08417 . Cited by: Appendix B , §2 .

Yan et al. (2026) J. Yan, Y. Li, Z. Hu, Z. Wang, G. Cui, X. Qu, Y. Cheng, and Y. Zhang Learning to reason under off-policy guidance . In The Thirty-ninth Annual Conference on Neural Information Processing Systems , External Links: Link Cited by: §C.1 .

Yang et al. (2025) A. Yang, A. Li, B. Yang, B. Zhang, B. Hui, B. Zheng, B. Yu, C. Gao, C. Huang, C. Lv, C. Zheng, D. Liu, F. Zhou, F. Huang, F. Hu, H. Ge, H. Wei, H. Lin, J. Tang, J. Yang, J. Tu, J. Zhang, J. Yang, J. Yang, J. Zhou, J. Zhou, J. Lin, K. Dang, K. Bao, K. Yang, L. Yu, L. Deng, M. Li, M. Xue, M. Li, P. Zhang, P. Wang, Q. Zhu, R. Men, R. Gao, S. Liu, S. Luo, T. Li, T. Tang, W. Yin, X. Ren, X. Wang, X. Zhang, X. Ren, Y. Fan, Y. Su, Y. Zhang, Y. Zhang, Y. Wan, Y. Liu, Z. Wang, Z. Cui, Z. Zhang, Z. Zhou, and Z. Qiu Qwen3 technical report . External Links: 2505.09388 , Link Cited by: §5.1 .

Yi et al. (2025) J. Yi, J. Wang, and S. Li ShorterBetter: guiding reasoning models to find optimal inference length for efficient reasoning . In The Thirty-ninth Annual Conference on Neural Information Processing Systems , External Links: Link Cited by: Appendix B , §1 , §2 .

Yu et al. (2025) Q. Yu, Z. Zhang, R. Zhu, Y. Yuan, X. Zuo, YuYue, W. Dai, T. Fan, G. Liu, J. Liu, L. Liu, X. Liu, H. Lin, Z. Lin, B. Ma, G. Sheng, Y. Tong, C. Zhang, M. Zhang, R. Zhang, W. Zhang, H. Zhu, J. Zhu, J. Chen, J. Chen, C. Wang, H. Yu, Y. Song, X. Wei, H. Zhou, J. Liu, W. Ma, Y. Zhang, L. Yan, Y. Wu, and M. Wang DAPO: an open-source LLM reinforcement learning system at scale . In The Thirty-ninth Annual Conference on Neural Information Processing Systems , External Links: Link Cited by: Appendix B , §1 , §2 , §5.1 .

Zhang et al. (2026a) F. Zhang, Z. Tan, X. Ma, Z. Dong, X. Leng, J. Zhao, X. Sun, and Y. Yang ADHint: adaptive hints with difficulty priors for reinforcement learning . External Links: 2512.13095 , Link Cited by: §C.1 .

Zhang et al. (2026b) L. Zhang, D. Song, Z. Wu, Z. Chen, C. Zhang, Y. Tian, H. Ma, C. Li, C. Zhou, X. Li, et al. PruneTIR: inference-time tool call pruning for effective yet efficient tool-integrated reasoning . arXiv preprint arXiv:2605.09931 . Cited by: §1 .

Zhang et al. (2025) L. Zhang, D. Song, Z. Wu, Y. Tian, C. Zhou, J. Xu, Z. Yang, and S. Zhang Detecting hallucination in large language models through deep internal representation analysis . In Proceedings of the Thirty-Fourth International Joint Conference on Artificial Intelligence, IJCAI-25 , pp. 8357–8365 . Cited by: §C.1 .

Zhao et al. (2026) Y. Zhao, Y. Liu, J. Liu, J. Chen, X. Wu, Y. Hao, T. Lv, S. Huang, L. Cui, Q. Ye, F. Wan, and F. Wei Geometric-mean policy optimization . In The Fourteenth International Conference on Learning Representations , External Links: Link Cited by: Appendix B , §1 , §2 , §5.1 , §5.1 .

Zheng et al. (2025) C. Zheng, S. Liu, M. Li, X. Chen, B. Yu, C. Gao, K. Dang, Y. Liu, R. Men, A. Yang, J. Zhou, and J. Lin Group sequence policy optimization . External Links: 2507.18071 , Link Cited by: §1 .

## Appendix A Proofs

### A.1 Proof of Proposition 1 and Lemma 1

We first state two auxiliary lemmas.

###### Lemma 3 .

For any query q q with σ q > 0 \sigma_{q}>0 , define A + ​ ( o ∣ q ) = max ⁡ ( A ⁡ ( o ∣ q ) , 0 ) , A − ​ ( o ∣ q ) = max ⁡ ( − A ⁡ ( o ∣ q ) , 0 ) . A^{+}(o\mid q)=\max(A(o\mid q),0),\qquad A^{-}(o\mid q)=\max(-A(o\mid q),0). Let W + ( q ) = 𝔼 o ∼ π old ( ⋅ ∣ q ) [ A + ( o ∣ q ) ] , W − ( q ) = 𝔼 o ∼ π old ( ⋅ ∣ q ) [ A − ( o ∣ q ) ] . W_{+}(q)=\mathbb{E}_{o\sim\pi_{\rm old}(\cdot\mid q)}[A^{+}(o\mid q)],\qquad W_{-}(q)=\mathbb{E}_{o\sim\pi_{\rm old}(\cdot\mid q)}[A^{-}(o\mid q)]. Then W + ​ ( q ) = W − ​ ( q ) W_{+}(q)=W_{-}(q) . Denoting the common value by W ⁡ ( q ) W(q) , we have W ( q ) = 1 2 𝔼 o ¯ ∼ π old ( ⋅ ∣ q ) [ | A ( o ¯ ∣ q ) | ] > 0 . W(q)=\frac{1}{2}\mathbb{E}_{\bar{o}\sim\pi_{\rm old}(\cdot\mid q)}\big[|A(\bar{o}\mid q)|\big]>0. Moreover, π ~ q + ​ ( d ​ o ) = A + ​ ( o ∣ q ) W ⁡ ( q ) ​ π old ​ ( d ​ o ∣ q ) , π ~ q − ​ ( d ​ o ) = A − ​ ( o ∣ q ) W ⁡ ( q ) ​ π old ​ ( d ​ o ∣ q ) \widetilde{\pi}_{q}^{+}(do)=\frac{A^{+}(o\mid q)}{W(q)}\,\pi_{\rm old}(do\mid q),\qquad\widetilde{\pi}_{q}^{-}(do)=\frac{A^{-}(o\mid q)}{W(q)}\,\pi_{\rm old}(do\mid q) are valid probability distributions.

###### Proof.

By the definition of A ⁡ ( o ∣ q ) A(o\mid q) , 𝔼 o ∼ π old ( ⋅ ∣ q ) [ A ( o ∣ q ) ] \displaystyle\mathbb{E}_{o\sim\pi_{\rm old}(\cdot\mid q)}[A(o\mid q)] = 𝔼 o ∼ π old ( ⋅ ∣ q ) [ r ( o , q ) ] − μ q σ q = 0 . \displaystyle=\frac{\mathbb{E}_{o\sim\pi_{\rm old}(\cdot\mid q)}[r(o,q)]-\mu_{q}}{\sigma_{q}}=0. Since A = A + − A − A=A^{+}-A^{-} , it follows that W + ​ ( q ) = W − ​ ( q ) W_{+}(q)=W_{-}(q) . Since | A | = A + + A − |A|=A^{+}+A^{-} , 𝔼 o ¯ ∼ π old ( ⋅ ∣ q ) [ | A ( o ¯ ∣ q ) | ] = W + ( q ) + W − ( q ) = 2 W ( q ) . \mathbb{E}_{\bar{o}\sim\pi_{\rm old}(\cdot\mid q)}\big[|A(\bar{o}\mid q)|\big]=W_{+}(q)+W_{-}(q)=2W(q). The condition σ q > 0 \sigma_{q}>0 implies that A ⁡ ( o ∣ q ) A(o\mid q) is not almost surely zero; combined with 𝔼 ⁡ [ A ⁡ ( o ∣ q ) ] = 0 \mathbb{E}[A(o\mid q)]=0 , this gives W ⁡ ( q ) > 0 W(q)>0 .

The reweighted measures are nonnegative, and ∫ π ~ q + ​ ( 𝑑 o ) = 𝔼 o ∼ π old ( ⋅ ∣ q ) [ A + ( o ∣ q ) ] W ⁡ ( q ) = 1 , ∫ π ~ q − ​ ( 𝑑 o ) = 𝔼 o ∼ π old ( ⋅ ∣ q ) [ A − ( o ∣ q ) ] W ⁡ ( q ) = 1 . \int\widetilde{\pi}_{q}^{+}(do)=\frac{\mathbb{E}_{o\sim\pi_{\rm old}(\cdot\mid q)}[A^{+}(o\mid q)]}{W(q)}=1,\qquad\int\widetilde{\pi}_{q}^{-}(do)=\frac{\mathbb{E}_{o\sim\pi_{\rm old}(\cdot\mid q)}[A^{-}(o\mid q)]}{W(q)}=1. ∎

###### Lemma 4 .

Assume binary rewards r ⁡ ( o , q ) ∈ { 0 , 1 } r(o,q)\in\{0,1\} and define p ( q ) = 𝔼 o ∼ π old ( ⋅ ∣ q ) [ r ( o , q ) ] . p(q)=\mathbb{E}_{o\sim\pi_{\rm old}(\cdot\mid q)}[r(o,q)]. For any query with 0 < p ⁡ ( q ) < 1 0<p(q)<1 , W ( q ) = p ​ ( q ) ​ ( 1 − p ​ ( q ) ) , π ~ q + = π old + ( ⋅ ∣ q ) , π ~ q − = π old − ( ⋅ ∣ q ) , W(q)=\sqrt{p(q)(1-p(q))},\qquad\widetilde{\pi}_{q}^{+}=\pi_{\rm old}^{+}(\cdot\mid q),\qquad\widetilde{\pi}_{q}^{-}=\pi_{\rm old}^{-}(\cdot\mid q), where π old + ( ⋅ ∣ q ) \pi_{\rm old}^{+}(\cdot\mid q) and π old − ( ⋅ ∣ q ) \pi_{\rm old}^{-}(\cdot\mid q) denote π old \pi_{\rm old} conditioned on r ⁡ ( o , q ) = 1 r(o,q)=1 and r ⁡ ( o , q ) = 0 r(o,q)=0 , respectively.

###### Proof.

Under binary rewards, μ q = p ⁡ ( q ) , σ q = p ​ ( q ) ​ ( 1 − p ​ ( q ) ) , A ⁡ ( o ∣ q ) = { 1 − p ⁡ ( q ) p ⁡ ( q ) , r ⁡ ( o , q ) = 1 , − p ⁡ ( q ) 1 − p ⁡ ( q ) , r ⁡ ( o , q ) = 0 . \mu_{q}=p(q),\qquad\sigma_{q}=\sqrt{p(q)(1-p(q))},\qquad A(o\mid q)=\begin{cases}\sqrt{\dfrac{1-p(q)}{p(q)}},&r(o,q)=1,\\[8.0pt] -\sqrt{\dfrac{p(q)}{1-p(q)}},&r(o,q)=0.\end{cases} Therefore, W ( q ) = 𝔼 o ∼ π old ( ⋅ ∣ q ) [ A + ( o ∣ q ) ] = p ( q ) 1 − p ⁡ ( q ) p ⁡ ( q ) = p ​ ( q ) ​ ( 1 − p ​ ( q ) ) . W(q)=\mathbb{E}_{o\sim\pi_{\rm old}(\cdot\mid q)}[A^{+}(o\mid q)]=p(q)\sqrt{\frac{1-p(q)}{p(q)}}=\sqrt{p(q)(1-p(q))}. Substituting this value into the definitions of π ~ q + \widetilde{\pi}_{q}^{+} and π ~ q − \widetilde{\pi}_{q}^{-} gives π ~ q + ​ ( d ​ o ) \displaystyle\widetilde{\pi}_{q}^{+}(do) = A + ​ ( o ∣ q ) W ⁡ ( q ) ​ π old ​ ( d ​ o ∣ q ) = 𝟏 { r ( o , q ) = 1 } p ⁡ ( q ) ​ π old ​ ( d ​ o ∣ q ) = π old + ​ ( d ​ o ∣ q ) , \displaystyle=\frac{A^{+}(o\mid q)}{W(q)}\,\pi_{\rm old}(do\mid q)=\frac{\mathbf{1}\{r(o,q)=1\}}{p(q)}\,\pi_{\rm old}(do\mid q)=\pi_{\rm old}^{+}(do\mid q), π ~ q − ​ ( d ​ o ) \displaystyle\widetilde{\pi}_{q}^{-}(do) = A − ​ ( o ∣ q ) W ⁡ ( q ) ​ π old ​ ( d ​ o ∣ q ) = 𝟏 { r ( o , q ) = 0 } 1 − p ⁡ ( q ) ​ π old ​ ( d ​ o ∣ q ) = π old − ​ ( d ​ o ∣ q ) . \displaystyle=\frac{A^{-}(o\mid q)}{W(q)}\,\pi_{\rm old}(do\mid q)=\frac{\mathbf{1}\{r(o,q)=0\}}{1-p(q)}\,\pi_{\rm old}(do\mid q)=\pi_{\rm old}^{-}(do\mid q). ∎

###### Proof of Proposition 1 .

Let ρ t ​ ( θ ) = π θ ​ ( o t ∣ q , o < t ) π old ​ ( o t ∣ q , o < t ) . \rho_{t}(\theta)=\frac{\pi_{\theta}(o_{t}\mid q,o_{<t})}{\pi_{\rm old}(o_{t}\mid q,o_{<t})}. Removing the KL regularizer from Eq. ( 2 ) gives 𝒥 GRPO clip ( θ ) = 𝔼 q 𝔼 o ∼ π old ( ⋅ ∣ q ) [ 1 | o | ∑ t = 1 | o | f ( ρ t ( θ ) , A ( o ∣ q ) ) ] . \mathcal{J}_{\rm GRPO}^{\rm clip}(\theta)=\mathbb{E}_{q}\mathbb{E}_{o\sim\pi_{\rm old}(\cdot\mid q)}\left[\frac{1}{|o|}\sum_{t=1}^{|o|}f\left(\rho_{t}(\theta),A(o\mid q)\right)\right]. For the clipping function in Eq. ( 3 ), f ⁡ ( ρ , A ⁡ ( o ∣ q ) ) = A + ​ ( o ∣ q ) ​ min ​ ( ρ , 1 + ϵ ) − A − ​ ( o ∣ q ) ​ max ​ ( ρ , 1 − ϵ ) . f(\rho,A(o\mid q))=A^{+}(o\mid q)\min(\rho,1+\epsilon)-A^{-}(o\mid q)\max(\rho,1-\epsilon). Averaging over tokens and using Eq. ( 6 ), we obtain 𝒥 GRPO clip ​ ( θ ) \displaystyle\mathcal{J}_{\rm GRPO}^{\rm clip}(\theta) = 𝔼 q 𝔼 o ∼ π old ( ⋅ ∣ q ) [ A + ( o ∣ q ) s θ + ( o , q ) − A − ( o ∣ q ) s θ − ( o , q ) ] \displaystyle=\mathbb{E}_{q}\mathbb{E}_{o\sim\pi_{\rm old}(\cdot\mid q)}\left[A^{+}(o\mid q)s_{\theta}^{+}(o,q)-A^{-}(o\mid q)s_{\theta}^{-}(o,q)\right] = 𝔼 q ​ W ​ ( q ) ​ ( 𝔼 o ∼ π ~ q + ​ [ s θ + ​ ( o , q ) ] − 𝔼 o ′ ∼ π ~ q − ​ [ s θ − ​ ( o ′ , q ) ] ) \displaystyle=\mathbb{E}_{q}W(q)\left(\mathbb{E}_{o\sim\widetilde{\pi}_{q}^{+}}[s_{\theta}^{+}(o,q)]-\mathbb{E}_{o^{\prime}\sim\widetilde{\pi}_{q}^{-}}[s_{\theta}^{-}(o^{\prime},q)]\right) = 𝔼 q ​ W ​ ( q ) ​ 𝔼 o ∼ π ~ q + , o ′ ∼ π ~ q − ​ [ s θ + ​ ( o , q ) − s θ − ​ ( o ′ , q ) ] . \displaystyle=\mathbb{E}_{q}W(q)\,\mathbb{E}_{o\sim\widetilde{\pi}_{q}^{+},\,o^{\prime}\sim\widetilde{\pi}_{q}^{-}}\left[s_{\theta}^{+}(o,q)-s_{\theta}^{-}(o^{\prime},q)\right]. (14) By Lemma 3 , W ( q ) = 1 2 𝔼 o ¯ ∼ π old ( ⋅ ∣ q ) [ | A ( o ¯ ∣ q ) | ] . W(q)=\frac{1}{2}\mathbb{E}_{\bar{o}\sim\pi_{\rm old}(\cdot\mid q)}\big[|A(\bar{o}\mid q)|\big]. Substituting this identity into Eq. ( 14 ) yields 𝒥 GRPO clip ( θ ) = 𝔼 q [ 1 2 𝔼 o ¯ ∼ π old ( ⋅ ∣ q ) [ | A ( o ¯ ∣ q ) | ] 𝔼 o ∼ π ~ q + , o ′ ∼ π ~ q − [ s θ + ( o , q ) − s θ − ( o ′ , q ) ] ] , \displaystyle\mathcal{J}_{\rm GRPO}^{\rm clip}(\theta)=\mathbb{E}_{q}\left[\frac{1}{2}\mathbb{E}_{\bar{o}\sim\pi_{\rm old}(\cdot\mid q)}\big[|A(\bar{o}\mid q)|\big]\,\mathbb{E}_{o\sim\widetilde{\pi}_{q}^{+},\,o^{\prime}\sim\widetilde{\pi}_{q}^{-}}\big[s_{\theta}^{+}(o,q)-s_{\theta}^{-}(o^{\prime},q)\big]\right], which is Eq. ( 4 ).

For binary rewards with 0 < p ⁡ ( q ) < 1 0<p(q)<1 , applying Lemma 4 to Eq. ( 14 ) gives 𝒥 GRPO clip ( θ ) = 𝔼 q p ​ ( q ) ​ ( 1 − p ​ ( q ) ) 𝔼 o ∼ π old + ( ⋅ ∣ q ) , o ′ ∼ π old − ( ⋅ ∣ q ) [ s θ + ( o , q ) − s θ − ( o ′ , q ) ] , \displaystyle\mathcal{J}_{\rm GRPO}^{\rm clip}(\theta)=\mathbb{E}_{q}\sqrt{p(q)(1-p(q))}\,\mathbb{E}_{o\sim\pi_{\rm old}^{+}(\cdot\mid q),\,o^{\prime}\sim\pi_{\rm old}^{-}(\cdot\mid q)}\big[s_{\theta}^{+}(o,q)-s_{\theta}^{-}(o^{\prime},q)\big], which is Eq. ( 5 ). ∎

###### Proof of Lemma 1 .

By Eq. ( 7 ), 𝒥 ^ GRPO clip ​ ( q ) = p ^ q ​ ( 1 − p ^ q ) ​ ( 1 N + ​ ∑ i = 1 N + s θ + ​ ( o i + , q ) − 1 N − ​ ∑ j = 1 N − s θ − ​ ( o j − , q ) ) . \widehat{\mathcal{J}}_{\rm GRPO}^{\rm clip}(q)=\sqrt{\widehat{p}_{q}(1-\widehat{p}_{q})}\left(\frac{1}{N^{+}}\sum_{i=1}^{N^{+}}s_{\theta}^{+}(o_{i}^{+},q)-\frac{1}{N^{-}}\sum_{j=1}^{N^{-}}s_{\theta}^{-}(o_{j}^{-},q)\right). Differentiating with respect to s θ + ​ ( o i + , q ) s_{\theta}^{+}(o_{i}^{+},q) and s θ − ​ ( o j − , q ) s_{\theta}^{-}(o_{j}^{-},q) gives ∂ 𝒥 ^ GRPO clip ​ ( q ) ∂ s θ + ​ ( o i + , q ) = 1 N + ​ p ^ q ​ ( 1 − p ^ q ) , ∂ 𝒥 ^ GRPO clip ​ ( q ) ∂ s θ − ​ ( o j − , q ) = − 1 N − ​ p ^ q ​ ( 1 − p ^ q ) . \frac{\partial\widehat{\mathcal{J}}_{\rm GRPO}^{\rm clip}(q)}{\partial s_{\theta}^{+}(o_{i}^{+},q)}=\frac{1}{N^{+}}\sqrt{\widehat{p}_{q}(1-\widehat{p}_{q})},\qquad\frac{\partial\widehat{\mathcal{J}}_{\rm GRPO}^{\rm clip}(q)}{\partial s_{\theta}^{-}(o_{j}^{-},q)}=-\frac{1}{N^{-}}\sqrt{\widehat{p}_{q}(1-\widehat{p}_{q})}. ∎

### A.2 Proof of Lemma 2

###### Proof.

For brevity, write s i + = s θ ​ ( o i + , q ) , s j − = s θ ​ ( o j − , q ) , Z i = exp ⁡ ( s i + / τ ) + ∑ k = 1 N − exp ⁡ ( s k − / τ ) . s_{i}^{+}=s_{\theta}(o_{i}^{+},q),\qquad s_{j}^{-}=s_{\theta}(o_{j}^{-},q),\qquad Z_{i}=\exp(s_{i}^{+}/\tau)+\sum_{k=1}^{N^{-}}\exp(s_{k}^{-}/\tau). Then Eq. ( 8 ) becomes 𝒥 ^ NCE ​ ( q ) = 1 N + ​ ∑ i = 1 N + τ ⁡ ( s i + τ − log ⁡ Z i ) , \widehat{\mathcal{J}}_{\rm NCE}(q)=\frac{1}{N^{+}}\sum_{i=1}^{N^{+}}\tau\left(\frac{s_{i}^{+}}{\tau}-\log Z_{i}\right), and the contrastive probabilities in Eq. ( 9 ) are P i + = exp ⁡ ( s i + / τ ) Z i , P i ​ j − = exp ⁡ ( s j − / τ ) Z i . P_{i}^{+}=\frac{\exp(s_{i}^{+}/\tau)}{Z_{i}},\qquad P_{ij}^{-}=\frac{\exp(s_{j}^{-}/\tau)}{Z_{i}}.

For a positive rollout score s i + s_{i}^{+} , only the i i -th summand depends on it. Hence ∂ 𝒥 ^ NCE ​ ( q ) ∂ s i + \displaystyle\frac{\partial\widehat{\mathcal{J}}_{\rm NCE}(q)}{\partial s_{i}^{+}} = 1 N + ​ τ ​ ( 1 τ − 1 Z i ​ ∂ Z i ∂ s i + ) = 1 N + ​ τ ​ ( 1 τ − 1 Z i ​ exp ⁡ ( s i + / τ ) τ ) = 1 N + ​ ( 1 − P i + ) . \displaystyle=\frac{1}{N^{+}}\tau\left(\frac{1}{\tau}-\frac{1}{Z_{i}}\frac{\partial Z_{i}}{\partial s_{i}^{+}}\right)=\frac{1}{N^{+}}\tau\left(\frac{1}{\tau}-\frac{1}{Z_{i}}\frac{\exp(s_{i}^{+}/\tau)}{\tau}\right)=\frac{1}{N^{+}}(1-P_{i}^{+}). For a negative rollout score s j − s_{j}^{-} , all summands depend on it through their denominators, giving ∂ 𝒥 ^ NCE ​ ( q ) ∂ s j − \displaystyle\frac{\partial\widehat{\mathcal{J}}_{\rm NCE}(q)}{\partial s_{j}^{-}} = − 1 N + ∑ i = 1 N + τ 1 Z i ∂ Z i ∂ s j − = − 1 N + ∑ i = 1 N + τ 1 Z i exp ⁡ ( s j − / τ ) τ = − 1 N + ∑ i = 1 N + P i ​ j − . \displaystyle=-\frac{1}{N^{+}}\sum_{i=1}^{N^{+}}\tau\frac{1}{Z_{i}}\frac{\partial Z_{i}}{\partial s_{j}^{-}}=-\frac{1}{N^{+}}\sum_{i=1}^{N^{+}}\tau\frac{1}{Z_{i}}\frac{\exp(s_{j}^{-}/\tau)}{\tau}=-\frac{1}{N^{+}}\sum_{i=1}^{N^{+}}P_{ij}^{-}. This gives Eq. ( 10 ).

Using P i + + ∑ j = 1 N − P i ​ j − = 1 P_{i}^{+}+\sum_{j=1}^{N^{-}}P_{ij}^{-}=1 , we obtain ∑ j = 1 N − ∂ 𝒥 ^ NCE ​ ( q ) ∂ s j − \displaystyle\sum_{j=1}^{N^{-}}\frac{\partial\widehat{\mathcal{J}}_{\rm NCE}(q)}{\partial s_{j}^{-}} = − 1 N + ∑ i = 1 N + ∑ j = 1 N − P i ​ j − = − 1 N + ∑ i = 1 N + ( 1 − P i + ) . \displaystyle=-\frac{1}{N^{+}}\sum_{i=1}^{N^{+}}\sum_{j=1}^{N^{-}}P_{ij}^{-}=-\frac{1}{N^{+}}\sum_{i=1}^{N^{+}}(1-P_{i}^{+}). Moreover, ∑ i = 1 N + ∂ 𝒥 ^ NCE ​ ( q ) ∂ s i + + ∑ j = 1 N − ∂ 𝒥 ^ NCE ​ ( q ) ∂ s j − \displaystyle\sum_{i=1}^{N^{+}}\frac{\partial\widehat{\mathcal{J}}_{\rm NCE}(q)}{\partial s_{i}^{+}}+\sum_{j=1}^{N^{-}}\frac{\partial\widehat{\mathcal{J}}_{\rm NCE}(q)}{\partial s_{j}^{-}} = 1 N + ​ ∑ i = 1 N + ( 1 − P i + ) − 1 N + ​ ∑ i = 1 N + ( 1 − P i + ) = 0 . \displaystyle=\frac{1}{N^{+}}\sum_{i=1}^{N^{+}}(1-P_{i}^{+})-\frac{1}{N^{+}}\sum_{i=1}^{N^{+}}(1-P_{i}^{+})=0. This gives Eq. ( 11 ).

Finally, for any two negative rollouts o j − o_{j}^{-} and o k − o_{k}^{-} , P i ​ j − \displaystyle P_{ij}^{-} = exp ⁡ ( s j − / τ ) Z i = exp ⁡ ( s j − − s k − τ ) ​ exp ⁡ ( s k − / τ ) Z i = exp ⁡ ( s j − − s k − τ ) ​ P i ​ k − . \displaystyle=\frac{\exp(s_{j}^{-}/\tau)}{Z_{i}}=\exp\left(\frac{s_{j}^{-}-s_{k}^{-}}{\tau}\right)\frac{\exp(s_{k}^{-}/\tau)}{Z_{i}}=\exp\left(\frac{s_{j}^{-}-s_{k}^{-}}{\tau}\right)P_{ik}^{-}. Summing over i i yields ∑ i = 1 N + P i ​ j − = exp ⁡ ( s j − − s k − τ ) ​ ∑ i = 1 N + P i ​ k − . \sum_{i=1}^{N^{+}}P_{ij}^{-}=\exp\left(\frac{s_{j}^{-}-s_{k}^{-}}{\tau}\right)\sum_{i=1}^{N^{+}}P_{ik}^{-}. Combining this identity with Eq. ( 10 ) gives | ∂ 𝒥 ^ NCE ​ ( q ) / ∂ s j − | | ∂ 𝒥 ^ NCE ​ ( q ) / ∂ s k − | = ∑ i = 1 N + P i ​ j − ∑ i = 1 N + P i ​ k − = exp ⁡ ( s j − − s k − τ ) . \frac{\left|\partial\widehat{\mathcal{J}}_{\rm NCE}(q)/\partial s_{j}^{-}\right|}{\left|\partial\widehat{\mathcal{J}}_{\rm NCE}(q)/\partial s_{k}^{-}\right|}=\frac{\sum_{i=1}^{N^{+}}P_{ij}^{-}}{\sum_{i=1}^{N^{+}}P_{ik}^{-}}=\exp\left(\frac{s_{j}^{-}-s_{k}^{-}}{\tau}\right). This gives Eq. ( 12 ). ∎

## Appendix B Detailed Related Work

#### RLVR for LLM reasoning.

GRPO ( 11 ) circumvents the reliance on a parameterized value model by estimating group-relative advantages directly from outcome-level rewards, maintaining robust performance across diverse reasoning tasks. Building on this paradigm, subsequent efforts have advanced GRPO toward better performance, more stable training, and more efficient reasoning. Dr. GRPO ( 20 ) removes the standard deviation in advantage estimation to mitigate difficulty bias, and adjusts loss allocation to reduce length bias. DAPO ( 35 ) introduces dynamic sampling to reduce high-variance gradient updates, and employs length-based reward shaping for finer-grained length control. GMPO ( 39 ) addresses outlier effects in asynchronous training by replacing token-level clipping with sequence-level geometric-mean clipping. CISPO ( 4 ) introduces soft clipping to preserve learning signals discarded by hard clipping, while SAPO ( 9 ) applies asymmetric, temperature-controlled clipping for finer-grained and stable updates. HAPO ( 15 ) and ShorterBetter ( 34 ) promote concise reasoning by selecting shorter trajectories from either historical or current generations. While DisCO ( 18 ) reinterprets GRPO through a discriminative lens under binary rewards and explores alternative scoring functions, it does not address the score-insensitive credit assignment induced by the GRPO objective. In contrast, ConSPO optimizes a group-wise InfoNCE-style objective for online RLVR, using likelihood-aligned sequence scores and a curriculum-scheduled margin to enable contrast-sensitive credit assignment within rollout groups.

#### Contrastive Objectives for Policy Alignment.

Contrastive learning trains models by distinguishing positive samples from negative distractors, with InfoNCE-style objectives serving as a standard formulation ( 24 ) . Foundational methods improve contrastive learning by enlarging the negative set, for example, through large-batch sampling in SimCLR ( 6 ) or momentum queues in MoCo ( 13 ) . Inspired by contrastive learning principles, recent policy alignment methods formulate preference optimization as a task of discriminating between preferred and dispreferred responses. For example, DPO ( 25 ) and CPO ( 31 ) optimize policy likelihoods over preference pairs without explicit reward modeling, while SimPO ( 22 ) further uses length-normalized sequence log-probabilities as an implicit ranking score. Unlike these offline preference optimization methods, ConSPO transfers the contrastive principle to online RLVR, where positive-negative comparisons are constructed from verifiable rewards on policy-generated rollouts rather than from static pairwise preference data.

## Appendix C Detailed Experimental Settings

### C.1 Benchmarks

We evaluate the mathematical reasoning ability of the trained models on seven benchmarks that are commonly used in recent RLVR studies ( 28 ; 32 ; 36 ; 38 ) .

• AIME24–26 — Problems from the 2024, 2025, and 2026 American Invitational Mathematics Examination. AIME consists of challenging high-school competition problems with integer-valued answers, requiring precise multi-step reasoning rather than multiple-choice selection.

• HMMT25 — Problems from the February 2025 Harvard–MIT Mathematics Tournament. The benchmark covers competition-level problems in algebra and number theory, combinatorics, and geometry, as well as team-based rounds.

• MATH500 — A 500-problem evaluation subset of the MATH benchmark. It contains competition-style problems across seven subjects, including algebra, geometry, counting and probability, number theory, and precalculus.

• AMC — Problems from the American Mathematics Competitions, covering a broad range of high-school mathematics topics. Although AMC is originally multiple-choice, the problems still require careful reasoning and symbolic manipulation.

• OlympiadBench — An Olympiad-level benchmark containing challenging mathematics and physics problems. It evaluates advanced scientific reasoning on problems collected from Olympiad-style competitions and related examinations.

Following prior work ( 21 ) , we report avg@32 on AIME, HMMT, and AMC to reduce the variance caused by their relatively small test sizes, and report pass@1 on the remaining benchmarks.

### C.2 Implementation Details

The training and evaluation configurations for the R1-Distill models are summarized in Tables 7 and 8 , respectively. For the experiments on Qwen3-4B-Base and Llama-3.2-3B-Instruct, we use the same settings as the R1-Distill models, except that the maximum response length is set to 4096 during both training and evaluation. In addition, Qwen3-4B-Base is trained for 2 epochs due to its faster convergence, while the other models are trained for 4 epochs.

## Appendix D Additional Experimental Results

To further evaluate the scalability of ConSPO, we conduct additional experiments on DeepSeek-R1-Distill-Qwen-32B. We use the same training data and hyperparameter settings as in the main experiments, and train both GRPO and ConSPO for two epochs. As shown in Table 9 , ConSPO outperforms GRPO across all seven benchmarks, improving the average score from 60.2 to 63.0. These results demonstrate that the effectiveness of ConSPO extends to substantially larger model scales.

## Appendix E Prompt Template

We use a unified prompt template for both rollout generation and evaluation across all methods. As shown in Figure 4 , the prompt asks the model to solve the problem step by step and place the final answer within \boxed{} , which facilitates answer extraction and rule-based verification.

## Instructions for reporting errors

We are continuing to improve HTML versions of papers, and your feedback helps enhance accessibility and mobile support. To report errors in the HTML that will help us improve conversion and rendering, choose any of the methods listed below:

Click the "Report Issue" ( ) button, located in the page header.

Tip: You can select the relevant text first, to include it in your report.

Our team has already identified the following issues . We appreciate your time reviewing and reporting rendering errors we may not have found yet. Your efforts will help us improve the HTML versions for all readers, because disability should not be a barrier to accessing research. Thank you for your continued support in championing open access for all.

Have a free development cycle? Help support accessibility at arXiv! Our collaborators at LaTeXML maintain a list of packages that need conversion , and welcome developer contributions .