Revisiting Reinforcement Learning with Verifiable Rewards from a Contrastive Perspective — part 2/4

Cleaned-text code points [15923, 31571). Bibliography omitted here; source.md includes it.

The proof of Lemma 1 follows by direct differentiation and is provided in Appendix A.1 .

#### Implications.

The reformulation identifies two objective-level limitations of GRPO. First, Eq. ( 6 ) reveals likelihood-misaligned surrogate scores : the positive-negative separation in GRPO is measured by sequence-level averages of clipped importance ratios, rather than by the sequence likelihoods that govern generation. Consequently, enlarging the surrogate score gap in Eq. ( 5 ) does not necessarily improve the likelihood ordering between positive and negative rollouts under the current policy. Second, Lemma 1 reveals score-insensitive credit assignment : rollout-level credit is determined by coefficients tied to the empirical pass rate p ^ q ​ ( 1 − p ^ q ) \sqrt{\widehat{p}_{q}(1-\widehat{p}_{q})} , rather than by current positive-negative score gaps within the group. As a result, all positives share the same update coefficient and all negatives share another, making the objective insensitive to whether a positive remains poorly separated from negatives or whether a negative is a high-scoring hard distractor.

### 4.2 A Contrastive View of RLVR Optimization

Motivated by the limitations above, ConSPO uses likelihood-aligned sequence scores and group-wise positive-negative contrasts. Specifically, it scores each rollout by the length-normalized log-probability s θ ​ ( o , q ) = 1 | o | ​ ∑ t = 1 | o | log ⁡ π θ ​ ( o t ∣ q , o < t ) s_{\theta}(o,q)=\frac{1}{|o|}\sum_{t=1}^{|o|}\log\pi_{\theta}(o_{t}\mid q,o_{<t}) . Unlike the clipped surrogate scores s θ + s_{\theta}^{+} and s θ − s_{\theta}^{-} in Eq. ( 6 ), the same scoring function is applied to both positive and negative rollouts, placing all rollouts on the likelihood scale that governs autoregressive generation. Given the rollout sets 𝒫 q \mathcal{P}_{q} and 𝒩 q \mathcal{N}_{q} , ConSPO treats negative rollouts as distractors and optimizes the following InfoNCE-style objective:

𝒥 ^ NCE ​ ( q ) = 1 N + ​ ∑ i = 1 N + τ \displaystyle\widehat{\mathcal{J}}_{\rm NCE}(q)=\frac{1}{N^{+}}\sum_{i=1}^{N^{+}}\tau (8) log ⁡ exp ⁡ ( s θ ​ ( o i + , q ) / τ ) exp ⁡ ( s θ ​ ( o i + , q ) / τ ) + ∑ j = 1 N − exp ⁡ ( s θ ​ ( o j − , q ) / τ ) . \displaystyle\log\frac{\exp(s_{\theta}(o_{i}^{+},q)/\tau)}{\exp(s_{\theta}(o_{i}^{+},q)/\tau)+\sum_{j=1}^{N^{-}}\exp(s_{\theta}(o_{j}^{-},q)/\tau)}. where τ > 0 \tau>0 is the temperature parameter. Unlike the linear positive-negative score difference in Eq. ( 5 ), Eq. ( 8 ) compares each positive rollout against the negative set via softmax normalization. For each positive rollout o i + o_{i}^{+} , we define the contrastive probabilities assigned to itself and to its negative distractors as

P i + \displaystyle P_{i}^{+} = exp ⁡ ( s θ ​ ( o i + , q ) / τ ) exp ⁡ ( s θ ​ ( o i + , q ) / τ ) + ∑ k = 1 N − exp ⁡ ( s θ ​ ( o k − , q ) / τ ) , \displaystyle=\frac{\exp(s_{\theta}(o_{i}^{+},q)/\tau)}{\exp(s_{\theta}(o_{i}^{+},q)/\tau)+\sum_{k=1}^{N^{-}}\exp(s_{\theta}(o_{k}^{-},q)/\tau)}, (9) P i ​ j − \displaystyle P_{ij}^{-} = exp ⁡ ( s θ ​ ( o j − , q ) / τ ) exp ⁡ ( s θ ​ ( o i + , q ) / τ ) + ∑ k = 1 N − exp ⁡ ( s θ ​ ( o k − , q ) / τ ) . \displaystyle=\frac{\exp(s_{\theta}(o_{j}^{-},q)/\tau)}{\exp(s_{\theta}(o_{i}^{+},q)/\tau)+\sum_{k=1}^{N^{-}}\exp(s_{\theta}(o_{k}^{-},q)/\tau)}.

By construction, P i + + ∑ j = 1 N − P i ​ j − = 1 P_{i}^{+}+\sum_{j=1}^{N^{-}}P_{ij}^{-}=1 for each positive rollout o i + o_{i}^{+} . These probabilities make rollout-level credit depend on within-group scores, as formalized in Lemma 2 .

###### Lemma 2 .

For the InfoNCE-style objective in Eq. ( 8 ), the derivatives with respect to the positive and negative rollout scores are ∂ 𝒥 ^ NCE ​ ( q ) ∂ s θ ​ ( o i + , q ) \displaystyle\frac{\partial\widehat{\mathcal{J}}_{\rm NCE}(q)}{\partial s_{\theta}(o_{i}^{+},q)} = 1 N + ​ ( 1 − P i + ) , \displaystyle=\frac{1}{N^{+}}(1-P_{i}^{+}), (10) ∂ 𝒥 ^ NCE ​ ( q ) ∂ s θ ​ ( o j − , q ) \displaystyle\frac{\partial\widehat{\mathcal{J}}_{\rm NCE}(q)}{\partial s_{\theta}(o_{j}^{-},q)} = − 1 N + ∑ i = 1 N + P i ​ j − . \displaystyle=-\frac{1}{N^{+}}\sum_{i=1}^{N^{+}}P_{ij}^{-}. Moreover, the aggregate derivative over the negative rollout scores and the total derivative over the group satisfy ∑ j = 1 N − ∂ 𝒥 ^ NCE ​ ( q ) ∂ s θ ​ ( o j − , q ) = − 1 N + ∑ i = 1 N + ( 1 − P i + ) , \displaystyle\sum_{j=1}^{N^{-}}\frac{\partial\widehat{\mathcal{J}}_{\rm NCE}(q)}{\partial s_{\theta}(o_{j}^{-},q)}=-\frac{1}{N^{+}}\sum_{i=1}^{N^{+}}(1-P_{i}^{+}), (11) ∑ i = 1 N + ∂ 𝒥 ^ NCE ​ ( q ) ∂ s θ ​ ( o i + , q ) + ∑ j = 1 N − ∂ 𝒥 ^ NCE ​ ( q ) ∂ s θ ​ ( o j − , q ) = 0 . \displaystyle\sum_{i=1}^{N^{+}}\frac{\partial\widehat{\mathcal{J}}_{\rm NCE}(q)}{\partial s_{\theta}(o_{i}^{+},q)}+\sum_{j=1}^{N^{-}}\frac{\partial\widehat{\mathcal{J}}_{\rm NCE}(q)}{\partial s_{\theta}(o_{j}^{-},q)}=0. For any two negative rollouts o j − o_{j}^{-} and o k − o_{k}^{-} , the magnitudes of their negative-score derivatives satisfy | ∂ 𝒥 ^ NCE ​ ( q ) / ∂ s θ ​ ( o j − , q ) | | ∂ 𝒥 ^ NCE ​ ( q ) / ∂ s θ ​ ( o k − , q ) | = ∑ i = 1 N + P i ​ j − ∑ i = 1 N + P i ​ k − \displaystyle\frac{\left|\partial\widehat{\mathcal{J}}_{\rm NCE}(q)/\partial s_{\theta}(o_{j}^{-},q)\right|}{\left|\partial\widehat{\mathcal{J}}_{\rm NCE}(q)/\partial s_{\theta}(o_{k}^{-},q)\right|}=\frac{\sum_{i=1}^{N^{+}}P_{ij}^{-}}{\sum_{i=1}^{N^{+}}P_{ik}^{-}} (12) = exp ⁡ ( s θ ​ ( o j − , q ) − s θ ​ ( o k − , q ) τ ) . \displaystyle=\exp\left(\frac{s_{\theta}(o_{j}^{-},q)-s_{\theta}(o_{k}^{-},q)}{\tau}\right).

The proof is provided in Appendix A.2 . Lemma 2 shows that ConSPO induces contrast-sensitive credit assignment. For positive rollouts, the coefficient 1 − P i + 1-P_{i}^{+} increases when the positive rollout remains close to its negative distractors, and attenuates as it becomes well separated. For negative rollouts, Eq. ( 11 ) shows that the aggregate suppressive signal has magnitude 1 N + ​ ∑ i = 1 N + ( 1 − P i + ) \frac{1}{N^{+}}\sum_{i=1}^{N^{+}}(1-P_{i}^{+}) , which grows when positives receive low contrastive probabilities. This signal is then allocated across negatives according to P i ​ j − P_{ij}^{-} , so higher-scoring negatives receive larger derivative magnitudes than lower-scoring ones, as shown in Eq. ( 12 ). The balance relation in Eq. ( 11 ) further shows that positive and negative score derivatives cancel at the group level. Compared with Lemma 1 , ConSPO assigns rollout-level credit according to current within-group score comparisons, rather than fixed coefficients determined by group reward statistics.

### 4.3 Scheduled Margin as a Separation Curriculum

Although the InfoNCE-style objective in Eq. ( 8 ) enables contrast-sensitive credit assignment, its gradients may attenuate once positive rollouts only marginally outperform their negative distractors. This becomes more pronounced in later training steps, where positives and negatives may be correctly ordered but remain insufficiently separated. To preserve separation pressure, ConSPO introduces a scheduled margin that lowers the effective score of each positive rollout in the contrastive objective. The margin starts from zero, allowing early training to focus on coarse positive-negative ordering, and gradually increases to enforce stronger separation as optimization progresses. At training step t t , ConSPO maximizes the resulting margin-enhanced contrastive objective:

𝒥 ^ ConSPO ​ ( q ) = 1 N + ​ ∑ i = 1 N + τ \displaystyle\widehat{\mathcal{J}}_{\rm ConSPO}(q)=\frac{1}{N^{+}}\sum_{i=1}^{N^{+}}\tau (13) log ⁡ exp ⁡ ( ( s θ ​ ( o i + , q ) − m t ) / τ ) exp ⁡ ( ( s θ ​ ( o i + , q ) − m t ) / τ ) + ∑ j = 1 N − exp ⁡ ( s θ ​ ( o j − , q ) / τ ) . \displaystyle\log\frac{\exp((s_{\theta}(o_{i}^{+},q)-m_{t})/\tau)}{\exp((s_{\theta}(o_{i}^{+},q)-m_{t})/\tau)+\sum_{j=1}^{N^{-}}\exp(s_{\theta}(o_{j}^{-},q)/\tau)}. where m t ≥ 0 m_{t}\geq 0 is the scheduled margin. By subtracting m t m_{t} from the positive score, ConSPO requires a positive rollout to exceed its negative distractors by a larger margin before the contrastive gradient attenuates.

Let λ t ∈ [ 0 , 1 ] \lambda_{t}\in[0,1] denote the normalized training progress, and let α ∈ ( 0 , 1 ] \alpha\in(0,1] denote the fraction of training used for margin warmup. We define the effective schedule progress and the margin as ρ t = min ⁡ ( λ t α , 1 ) , m t = M 2 ​ ( 1 − cos ⁡ ( π ​ ρ t ) ) . \rho_{t}=\min\left(\frac{\lambda_{t}}{\alpha},1\right),\quad m_{t}=\frac{M}{2}\left(1-\cos(\pi\rho_{t})\right). where M M is the target margin. Thus, the margin smoothly increases from zero to M M during the first α \alpha portion of training and remains fixed afterward.

## 5 Experiments

### 5.1 Settings

#### Models.

We conduct experiments across diverse model families and scales to validate the generalization of ConSPO, including DeepSeek-R1-Distill-Qwen-1.5B ( 11 ) , DeepSeek-R1-Distill-Qwen-7B, DeepSeek-R1-Distill-Llama-8B, Qwen3-4B-Base ( 33 ) , and Llama-3.2-3B-Instruct ( 10 ) .

#### Training data.

We use the DeepScaleR-Preview-Dataset ( 28 ) which comprises approximately 40k mathematical problems drawn from AIME ( 39 ) , AMC ( 19 ) , Omni-MATH ( 8 ) , and Still ( 23 ) . We also use DAPO-Math-17k ( 35 ) to validate the effectiveness of ConSPO on different training data.

#### Benchmarks.

We evaluate our models on seven benchmarks, including AIME 2024, AIME 2025, AIME 2026, HMMT 2025 ( 2 ) , MATH500 ( 14 ) , AMC, and OlympiadBench ( 12 ) . We report pass@1 on most benchmarks, and avg@32 on AIME, HMMT, and AMC to mitigate the variance introduced by the relatively small size of these datasets ( 21 ) . More details are provided in Appendix C.1 .

#### Baselines.

We compare ConSPO against seven recent state-of-the-art RLVR methods, including (1) GRPO ; (2) DAPO , which refines the GRPO loss formulation and introduces dynamic sampling and overlong reward shaping; (3) Dr. GRPO , which removes the standard deviation in advantage estimation and reorganizes loss allocation; (4) DisCO , which revisits GRPO from a discriminative perspective and eliminates its difficulty bias; (5) GMPO , which introduces sequence-level geometric-mean importance sampling; (6) CISPO , which introduces soft clipping in importance sampling; and (7) SAPO , which replaces hard clipping with a smooth, temperature-controlled gate to adaptively control off-policy updates.

#### Implementation details.

During training, we set the temperature to 1.0 to ensure sufficient exploration and the learning rate to 2 × 10 − 6 2\times 10^{-6} . The maximum response length is 8K for DeepSeek-R1-Distill-Qwen and DeepSeek-R1-Distill-Llama, and 4K for all other models. All methods run for 4 epochs, except for Qwen3-4B-Base, which runs for 2 epochs due to its faster convergence. Following standard practice ( 11 ; 39 ) , we discard all-positive and all-negative groups during training, as they do not permit a valid positive-negative contrast. We set the contrastive temperature τ \tau to 10, the target margin M M to 0.01, and the margin warmup ratio α \alpha to 30%. Evaluations are conducted every 100 steps, with a temperature of 0.6, a top- p p of 0.95, and the same maximum response length used during training. The best performance for each method is reported. Experiments on the 1.5B, 3B, and 4B models are run on a single compute node, whereas those on the 7B and 8B models are distributed across four compute nodes. More details are provided in Appendix C.2 .

### 5.2 Main Results

#### Performance with reasoning models.

As shown in Tables 1 and 2 , ConSPO delivers consistent improvements on reasoning models. On DeepSeek-R1-Distill-Qwen-1.5B, ConSPO achieves the best average performance of 44.4, outperforming GRPO by 4.4 points and the strongest baseline by 1.6 points. Moreover, ConSPO achieves the highest performance across all seven benchmarks in this setting, indicating its generalization across different reasoning tasks. The same trend holds for larger reasoning models. On DeepSeek-R1-Distill-Qwen-7B and DeepSeek-R1-Distill-Llama-8B, ConSPO improves over the strongest baseline by 2.1 and 1.9 points, respectively. These results suggest that the proposed contrastive optimization scheme remains effective across different parameter scales.

#### Generalization across backbone models and training datasets.

As shown in Tables 3 and 5 , ConSPO generalizes well across both base and instruction-tuned models. On Qwen3-4B-Base, ConSPO achieves the best average performance of 34.0, outperforming the strongest baseline by 2.1 points. Notably, on the challenging HMMT25 benchmark, where the base model obtains only 0.9, ConSPO improves the score to 8.0, suggesting that our method can substantially enhance performance on difficult reasoning tasks where the initial model performs poorly. On Llama-3.2-3B-Instruct, whose base reasoning capability is substantially weaker, ConSPO still achieves the highest average performance of 29.2, improving over the strongest baseline by 0.6 points. Moreover, Table 4 shows that ConSPO remains effective when trained on the DAPO-Math-17k dataset, achieving an average performance of 44.1 and outperforming the strongest baseline by 1.9 points. These results demonstrate that ConSPO’s effectiveness is not tied to a specific backbone or training dataset.

### 5.3 Ablation Study

Table 6 ablates the key components of ConSPO on DeepSeek-R1-Distill-Qwen-1.5B. Replacing the InfoNCE-style objective with a linear positive-negative score difference causes the largest performance drop, reducing the average score from 44.4 to 42.7. This result shows that group-wise contrastive optimization is central to ConSPO, as it enables credit assignment to adapt to current within-group score comparisons. Replacing the length-normalized sequence log-probabilities with clipped importance sampling ratio scores also decreases the average score to 43.4, confirming the importance of aligning rollout scores with the likelihood scale that governs autoregressive generation. The scheduled margin further improves performance: removing the margin reduces the average score to 44.1, while using a fixed margin achieves 43.9. These results indicate that the contrastive objective, likelihood-aligned sequence scoring, and curriculum-scheduled margin play complementary roles in ConSPO’s effectiveness.

### 5.4 Parameter Study

Figure 3 studies the sensitivity of ConSPO to the contrastive temperature τ \tau , target margin M M , and margin warmup ratio α \alpha . When τ \tau is too small, the contrastive distribution becomes overly sharp, potentially concentrating the update on a few high-scoring negatives and leading to less stable optimization. For the margin schedule, performance drops when either the target margin or the warmup ratio is large, suggesting that ConSPO benefits from a moderate separation requirement introduced at a proper pace.

## 6 Conclusion

In this paper, we derive an equivalent discriminative reformulation of GRPO as a weighted positive-negative discrimination objective. This reformulation exposes two objective-level limitations: likelihood-misaligned surrogate scores , where GRPO optimizes clipped ratio-based scores rather than generation likelihoods, and score-insensitive credit assignment , where rollout-level credit is insensitive to positive-negative score gaps. To address these limitations, we propose ConSPO, which optimizes a group-wise InfoNCE-style objective with likelihood-aligned sequence scores and a curriculum-scheduled margin. Extensive experiments across diverse settings validate the effectiveness and generalization of ConSPO.

## Limitations