Revisiting Reinforcement Learning with Verifiable Rewards from a Contrastive Perspective — part 3/4

Cleaned-text code points [31571, 47477). Bibliography omitted here; source.md includes it.

Despite our best efforts, this study has several limitations. First, although our experiments cover multiple backbone models, model scales, and training datasets, our empirical validation primarily focuses on models up to the 8B scale. While we additionally compare ConSPO and GRPO on a 32B model in Appendix D , more systematic evaluations at this scale and beyond remain an important direction for future work. Second, ConSPO is a preliminary exploration of RLVR from a contrastive optimization perspective. We instantiate this perspective with an InfoNCE-style group-wise objective, likelihood-aligned sequence scores, and a curriculum-scheduled margin. Broader choices of objective formulations and rollout scoring functions remain worth investigating to better understand and generalize contrast-sensitive credit assignment in RLVR.

## Ethical Considerations

This work studies reinforcement learning methods to improve LLM reasoning abilities and does not introduce new datasets that contain private, sensitive, or personally identifiable information. The potential negative impacts of ConSPO are similar to those associated with general LLM reasoning technologies, including risks from misuse or insufficiently supervised deployment. We emphasize the importance of careful evaluation and appropriate safeguards to ensure fair, safe, and responsible use of LLM systems.

## Acknowledgement

This work was supported by Qwen Business Unit through Alibaba Research Intern Program.

## Appendix A Proofs

### A.1 Proof of Proposition 1 and Lemma 1

We first state two auxiliary lemmas.

###### Lemma 3 .

For any query q q with σ q > 0 \sigma_{q}>0 , define A + ​ ( o ∣ q ) = max ⁡ ( A ⁡ ( o ∣ q ) , 0 ) , A − ​ ( o ∣ q ) = max ⁡ ( − A ⁡ ( o ∣ q ) , 0 ) . A^{+}(o\mid q)=\max(A(o\mid q),0),\qquad A^{-}(o\mid q)=\max(-A(o\mid q),0). Let W + ( q ) = 𝔼 o ∼ π old ( ⋅ ∣ q ) [ A + ( o ∣ q ) ] , W − ( q ) = 𝔼 o ∼ π old ( ⋅ ∣ q ) [ A − ( o ∣ q ) ] . W_{+}(q)=\mathbb{E}_{o\sim\pi_{\rm old}(\cdot\mid q)}[A^{+}(o\mid q)],\qquad W_{-}(q)=\mathbb{E}_{o\sim\pi_{\rm old}(\cdot\mid q)}[A^{-}(o\mid q)]. Then W + ​ ( q ) = W − ​ ( q ) W_{+}(q)=W_{-}(q) . Denoting the common value by W ⁡ ( q ) W(q) , we have W ( q ) = 1 2 𝔼 o ¯ ∼ π old ( ⋅ ∣ q ) [ | A ( o ¯ ∣ q ) | ] > 0 . W(q)=\frac{1}{2}\mathbb{E}_{\bar{o}\sim\pi_{\rm old}(\cdot\mid q)}\big[|A(\bar{o}\mid q)|\big]>0. Moreover, π ~ q + ​ ( d ​ o ) = A + ​ ( o ∣ q ) W ⁡ ( q ) ​ π old ​ ( d ​ o ∣ q ) , π ~ q − ​ ( d ​ o ) = A − ​ ( o ∣ q ) W ⁡ ( q ) ​ π old ​ ( d ​ o ∣ q ) \widetilde{\pi}_{q}^{+}(do)=\frac{A^{+}(o\mid q)}{W(q)}\,\pi_{\rm old}(do\mid q),\qquad\widetilde{\pi}_{q}^{-}(do)=\frac{A^{-}(o\mid q)}{W(q)}\,\pi_{\rm old}(do\mid q) are valid probability distributions.

###### Proof.

By the definition of A ⁡ ( o ∣ q ) A(o\mid q) , 𝔼 o ∼ π old ( ⋅ ∣ q ) [ A ( o ∣ q ) ] \displaystyle\mathbb{E}_{o\sim\pi_{\rm old}(\cdot\mid q)}[A(o\mid q)] = 𝔼 o ∼ π old ( ⋅ ∣ q ) [ r ( o , q ) ] − μ q σ q = 0 . \displaystyle=\frac{\mathbb{E}_{o\sim\pi_{\rm old}(\cdot\mid q)}[r(o,q)]-\mu_{q}}{\sigma_{q}}=0. Since A = A + − A − A=A^{+}-A^{-} , it follows that W + ​ ( q ) = W − ​ ( q ) W_{+}(q)=W_{-}(q) . Since | A | = A + + A − |A|=A^{+}+A^{-} , 𝔼 o ¯ ∼ π old ( ⋅ ∣ q ) [ | A ( o ¯ ∣ q ) | ] = W + ( q ) + W − ( q ) = 2 W ( q ) . \mathbb{E}_{\bar{o}\sim\pi_{\rm old}(\cdot\mid q)}\big[|A(\bar{o}\mid q)|\big]=W_{+}(q)+W_{-}(q)=2W(q). The condition σ q > 0 \sigma_{q}>0 implies that A ⁡ ( o ∣ q ) A(o\mid q) is not almost surely zero; combined with 𝔼 ⁡ [ A ⁡ ( o ∣ q ) ] = 0 \mathbb{E}[A(o\mid q)]=0 , this gives W ⁡ ( q ) > 0 W(q)>0 .

The reweighted measures are nonnegative, and ∫ π ~ q + ​ ( 𝑑 o ) = 𝔼 o ∼ π old ( ⋅ ∣ q ) [ A + ( o ∣ q ) ] W ⁡ ( q ) = 1 , ∫ π ~ q − ​ ( 𝑑 o ) = 𝔼 o ∼ π old ( ⋅ ∣ q ) [ A − ( o ∣ q ) ] W ⁡ ( q ) = 1 . \int\widetilde{\pi}_{q}^{+}(do)=\frac{\mathbb{E}_{o\sim\pi_{\rm old}(\cdot\mid q)}[A^{+}(o\mid q)]}{W(q)}=1,\qquad\int\widetilde{\pi}_{q}^{-}(do)=\frac{\mathbb{E}_{o\sim\pi_{\rm old}(\cdot\mid q)}[A^{-}(o\mid q)]}{W(q)}=1. ∎

###### Lemma 4 .

Assume binary rewards r ⁡ ( o , q ) ∈ { 0 , 1 } r(o,q)\in\{0,1\} and define p ( q ) = 𝔼 o ∼ π old ( ⋅ ∣ q ) [ r ( o , q ) ] . p(q)=\mathbb{E}_{o\sim\pi_{\rm old}(\cdot\mid q)}[r(o,q)]. For any query with 0 < p ⁡ ( q ) < 1 0<p(q)<1 , W ( q ) = p ​ ( q ) ​ ( 1 − p ​ ( q ) ) , π ~ q + = π old + ( ⋅ ∣ q ) , π ~ q − = π old − ( ⋅ ∣ q ) , W(q)=\sqrt{p(q)(1-p(q))},\qquad\widetilde{\pi}_{q}^{+}=\pi_{\rm old}^{+}(\cdot\mid q),\qquad\widetilde{\pi}_{q}^{-}=\pi_{\rm old}^{-}(\cdot\mid q), where π old + ( ⋅ ∣ q ) \pi_{\rm old}^{+}(\cdot\mid q) and π old − ( ⋅ ∣ q ) \pi_{\rm old}^{-}(\cdot\mid q) denote π old \pi_{\rm old} conditioned on r ⁡ ( o , q ) = 1 r(o,q)=1 and r ⁡ ( o , q ) = 0 r(o,q)=0 , respectively.

###### Proof.

Under binary rewards, μ q = p ⁡ ( q ) , σ q = p ​ ( q ) ​ ( 1 − p ​ ( q ) ) , A ⁡ ( o ∣ q ) = { 1 − p ⁡ ( q ) p ⁡ ( q ) , r ⁡ ( o , q ) = 1 , − p ⁡ ( q ) 1 − p ⁡ ( q ) , r ⁡ ( o , q ) = 0 . \mu_{q}=p(q),\qquad\sigma_{q}=\sqrt{p(q)(1-p(q))},\qquad A(o\mid q)=\begin{cases}\sqrt{\dfrac{1-p(q)}{p(q)}},&r(o,q)=1,\\[8.0pt] -\sqrt{\dfrac{p(q)}{1-p(q)}},&r(o,q)=0.\end{cases} Therefore, W ( q ) = 𝔼 o ∼ π old ( ⋅ ∣ q ) [ A + ( o ∣ q ) ] = p ( q ) 1 − p ⁡ ( q ) p ⁡ ( q ) = p ​ ( q ) ​ ( 1 − p ​ ( q ) ) . W(q)=\mathbb{E}_{o\sim\pi_{\rm old}(\cdot\mid q)}[A^{+}(o\mid q)]=p(q)\sqrt{\frac{1-p(q)}{p(q)}}=\sqrt{p(q)(1-p(q))}. Substituting this value into the definitions of π ~ q + \widetilde{\pi}_{q}^{+} and π ~ q − \widetilde{\pi}_{q}^{-} gives π ~ q + ​ ( d ​ o ) \displaystyle\widetilde{\pi}_{q}^{+}(do) = A + ​ ( o ∣ q ) W ⁡ ( q ) ​ π old ​ ( d ​ o ∣ q ) = 𝟏 { r ( o , q ) = 1 } p ⁡ ( q ) ​ π old ​ ( d ​ o ∣ q ) = π old + ​ ( d ​ o ∣ q ) , \displaystyle=\frac{A^{+}(o\mid q)}{W(q)}\,\pi_{\rm old}(do\mid q)=\frac{\mathbf{1}\{r(o,q)=1\}}{p(q)}\,\pi_{\rm old}(do\mid q)=\pi_{\rm old}^{+}(do\mid q), π ~ q − ​ ( d ​ o ) \displaystyle\widetilde{\pi}_{q}^{-}(do) = A − ​ ( o ∣ q ) W ⁡ ( q ) ​ π old ​ ( d ​ o ∣ q ) = 𝟏 { r ( o , q ) = 0 } 1 − p ⁡ ( q ) ​ π old ​ ( d ​ o ∣ q ) = π old − ​ ( d ​ o ∣ q ) . \displaystyle=\frac{A^{-}(o\mid q)}{W(q)}\,\pi_{\rm old}(do\mid q)=\frac{\mathbf{1}\{r(o,q)=0\}}{1-p(q)}\,\pi_{\rm old}(do\mid q)=\pi_{\rm old}^{-}(do\mid q). ∎

###### Proof of Proposition 1 .

Let ρ t ​ ( θ ) = π θ ​ ( o t ∣ q , o < t ) π old ​ ( o t ∣ q , o < t ) . \rho_{t}(\theta)=\frac{\pi_{\theta}(o_{t}\mid q,o_{<t})}{\pi_{\rm old}(o_{t}\mid q,o_{<t})}. Removing the KL regularizer from Eq. ( 2 ) gives 𝒥 GRPO clip ( θ ) = 𝔼 q 𝔼 o ∼ π old ( ⋅ ∣ q ) [ 1 | o | ∑ t = 1 | o | f ( ρ t ( θ ) , A ( o ∣ q ) ) ] . \mathcal{J}_{\rm GRPO}^{\rm clip}(\theta)=\mathbb{E}_{q}\mathbb{E}_{o\sim\pi_{\rm old}(\cdot\mid q)}\left[\frac{1}{|o|}\sum_{t=1}^{|o|}f\left(\rho_{t}(\theta),A(o\mid q)\right)\right]. For the clipping function in Eq. ( 3 ), f ⁡ ( ρ , A ⁡ ( o ∣ q ) ) = A + ​ ( o ∣ q ) ​ min ​ ( ρ , 1 + ϵ ) − A − ​ ( o ∣ q ) ​ max ​ ( ρ , 1 − ϵ ) . f(\rho,A(o\mid q))=A^{+}(o\mid q)\min(\rho,1+\epsilon)-A^{-}(o\mid q)\max(\rho,1-\epsilon). Averaging over tokens and using Eq. ( 6 ), we obtain 𝒥 GRPO clip ​ ( θ ) \displaystyle\mathcal{J}_{\rm GRPO}^{\rm clip}(\theta) = 𝔼 q 𝔼 o ∼ π old ( ⋅ ∣ q ) [ A + ( o ∣ q ) s θ + ( o , q ) − A − ( o ∣ q ) s θ − ( o , q ) ] \displaystyle=\mathbb{E}_{q}\mathbb{E}_{o\sim\pi_{\rm old}(\cdot\mid q)}\left[A^{+}(o\mid q)s_{\theta}^{+}(o,q)-A^{-}(o\mid q)s_{\theta}^{-}(o,q)\right] = 𝔼 q ​ W ​ ( q ) ​ ( 𝔼 o ∼ π ~ q + ​ [ s θ + ​ ( o , q ) ] − 𝔼 o ′ ∼ π ~ q − ​ [ s θ − ​ ( o ′ , q ) ] ) \displaystyle=\mathbb{E}_{q}W(q)\left(\mathbb{E}_{o\sim\widetilde{\pi}_{q}^{+}}[s_{\theta}^{+}(o,q)]-\mathbb{E}_{o^{\prime}\sim\widetilde{\pi}_{q}^{-}}[s_{\theta}^{-}(o^{\prime},q)]\right) = 𝔼 q ​ W ​ ( q ) ​ 𝔼 o ∼ π ~ q + , o ′ ∼ π ~ q − ​ [ s θ + ​ ( o , q ) − s θ − ​ ( o ′ , q ) ] . \displaystyle=\mathbb{E}_{q}W(q)\,\mathbb{E}_{o\sim\widetilde{\pi}_{q}^{+},\,o^{\prime}\sim\widetilde{\pi}_{q}^{-}}\left[s_{\theta}^{+}(o,q)-s_{\theta}^{-}(o^{\prime},q)\right]. (14) By Lemma 3 , W ( q ) = 1 2 𝔼 o ¯ ∼ π old ( ⋅ ∣ q ) [ | A ( o ¯ ∣ q ) | ] . W(q)=\frac{1}{2}\mathbb{E}_{\bar{o}\sim\pi_{\rm old}(\cdot\mid q)}\big[|A(\bar{o}\mid q)|\big]. Substituting this identity into Eq. ( 14 ) yields 𝒥 GRPO clip ( θ ) = 𝔼 q [ 1 2 𝔼 o ¯ ∼ π old ( ⋅ ∣ q ) [ | A ( o ¯ ∣ q ) | ] 𝔼 o ∼ π ~ q + , o ′ ∼ π ~ q − [ s θ + ( o , q ) − s θ − ( o ′ , q ) ] ] , \displaystyle\mathcal{J}_{\rm GRPO}^{\rm clip}(\theta)=\mathbb{E}_{q}\left[\frac{1}{2}\mathbb{E}_{\bar{o}\sim\pi_{\rm old}(\cdot\mid q)}\big[|A(\bar{o}\mid q)|\big]\,\mathbb{E}_{o\sim\widetilde{\pi}_{q}^{+},\,o^{\prime}\sim\widetilde{\pi}_{q}^{-}}\big[s_{\theta}^{+}(o,q)-s_{\theta}^{-}(o^{\prime},q)\big]\right], which is Eq. ( 4 ).

For binary rewards with 0 < p ⁡ ( q ) < 1 0<p(q)<1 , applying Lemma 4 to Eq. ( 14 ) gives 𝒥 GRPO clip ( θ ) = 𝔼 q p ​ ( q ) ​ ( 1 − p ​ ( q ) ) 𝔼 o ∼ π old + ( ⋅ ∣ q ) , o ′ ∼ π old − ( ⋅ ∣ q ) [ s θ + ( o , q ) − s θ − ( o ′ , q ) ] , \displaystyle\mathcal{J}_{\rm GRPO}^{\rm clip}(\theta)=\mathbb{E}_{q}\sqrt{p(q)(1-p(q))}\,\mathbb{E}_{o\sim\pi_{\rm old}^{+}(\cdot\mid q),\,o^{\prime}\sim\pi_{\rm old}^{-}(\cdot\mid q)}\big[s_{\theta}^{+}(o,q)-s_{\theta}^{-}(o^{\prime},q)\big], which is Eq. ( 5 ). ∎

###### Proof of Lemma 1 .

By Eq. ( 7 ), 𝒥 ^ GRPO clip ​ ( q ) = p ^ q ​ ( 1 − p ^ q ) ​ ( 1 N + ​ ∑ i = 1 N + s θ + ​ ( o i + , q ) − 1 N − ​ ∑ j = 1 N − s θ − ​ ( o j − , q ) ) . \widehat{\mathcal{J}}_{\rm GRPO}^{\rm clip}(q)=\sqrt{\widehat{p}_{q}(1-\widehat{p}_{q})}\left(\frac{1}{N^{+}}\sum_{i=1}^{N^{+}}s_{\theta}^{+}(o_{i}^{+},q)-\frac{1}{N^{-}}\sum_{j=1}^{N^{-}}s_{\theta}^{-}(o_{j}^{-},q)\right). Differentiating with respect to s θ + ​ ( o i + , q ) s_{\theta}^{+}(o_{i}^{+},q) and s θ − ​ ( o j − , q ) s_{\theta}^{-}(o_{j}^{-},q) gives ∂ 𝒥 ^ GRPO clip ​ ( q ) ∂ s θ + ​ ( o i + , q ) = 1 N + ​ p ^ q ​ ( 1 − p ^ q ) , ∂ 𝒥 ^ GRPO clip ​ ( q ) ∂ s θ − ​ ( o j − , q ) = − 1 N − ​ p ^ q ​ ( 1 − p ^ q ) . \frac{\partial\widehat{\mathcal{J}}_{\rm GRPO}^{\rm clip}(q)}{\partial s_{\theta}^{+}(o_{i}^{+},q)}=\frac{1}{N^{+}}\sqrt{\widehat{p}_{q}(1-\widehat{p}_{q})},\qquad\frac{\partial\widehat{\mathcal{J}}_{\rm GRPO}^{\rm clip}(q)}{\partial s_{\theta}^{-}(o_{j}^{-},q)}=-\frac{1}{N^{-}}\sqrt{\widehat{p}_{q}(1-\widehat{p}_{q})}. ∎

### A.2 Proof of Lemma 2

###### Proof.

For brevity, write s i + = s θ ​ ( o i + , q ) , s j − = s θ ​ ( o j − , q ) , Z i = exp ⁡ ( s i + / τ ) + ∑ k = 1 N − exp ⁡ ( s k − / τ ) . s_{i}^{+}=s_{\theta}(o_{i}^{+},q),\qquad s_{j}^{-}=s_{\theta}(o_{j}^{-},q),\qquad Z_{i}=\exp(s_{i}^{+}/\tau)+\sum_{k=1}^{N^{-}}\exp(s_{k}^{-}/\tau). Then Eq. ( 8 ) becomes 𝒥 ^ NCE ​ ( q ) = 1 N + ​ ∑ i = 1 N + τ ⁡ ( s i + τ − log ⁡ Z i ) , \widehat{\mathcal{J}}_{\rm NCE}(q)=\frac{1}{N^{+}}\sum_{i=1}^{N^{+}}\tau\left(\frac{s_{i}^{+}}{\tau}-\log Z_{i}\right), and the contrastive probabilities in Eq. ( 9 ) are P i + = exp ⁡ ( s i + / τ ) Z i , P i ​ j − = exp ⁡ ( s j − / τ ) Z i . P_{i}^{+}=\frac{\exp(s_{i}^{+}/\tau)}{Z_{i}},\qquad P_{ij}^{-}=\frac{\exp(s_{j}^{-}/\tau)}{Z_{i}}.

For a positive rollout score s i + s_{i}^{+} , only the i i -th summand depends on it. Hence ∂ 𝒥 ^ NCE ​ ( q ) ∂ s i + \displaystyle\frac{\partial\widehat{\mathcal{J}}_{\rm NCE}(q)}{\partial s_{i}^{+}} = 1 N + ​ τ ​ ( 1 τ − 1 Z i ​ ∂ Z i ∂ s i + ) = 1 N + ​ τ ​ ( 1 τ − 1 Z i ​ exp ⁡ ( s i + / τ ) τ ) = 1 N + ​ ( 1 − P i + ) . \displaystyle=\frac{1}{N^{+}}\tau\left(\frac{1}{\tau}-\frac{1}{Z_{i}}\frac{\partial Z_{i}}{\partial s_{i}^{+}}\right)=\frac{1}{N^{+}}\tau\left(\frac{1}{\tau}-\frac{1}{Z_{i}}\frac{\exp(s_{i}^{+}/\tau)}{\tau}\right)=\frac{1}{N^{+}}(1-P_{i}^{+}). For a negative rollout score s j − s_{j}^{-} , all summands depend on it through their denominators, giving ∂ 𝒥 ^ NCE ​ ( q ) ∂ s j − \displaystyle\frac{\partial\widehat{\mathcal{J}}_{\rm NCE}(q)}{\partial s_{j}^{-}} = − 1 N + ∑ i = 1 N + τ 1 Z i ∂ Z i ∂ s j − = − 1 N + ∑ i = 1 N + τ 1 Z i exp ⁡ ( s j − / τ ) τ = − 1 N + ∑ i = 1 N + P i ​ j − . \displaystyle=-\frac{1}{N^{+}}\sum_{i=1}^{N^{+}}\tau\frac{1}{Z_{i}}\frac{\partial Z_{i}}{\partial s_{j}^{-}}=-\frac{1}{N^{+}}\sum_{i=1}^{N^{+}}\tau\frac{1}{Z_{i}}\frac{\exp(s_{j}^{-}/\tau)}{\tau}=-\frac{1}{N^{+}}\sum_{i=1}^{N^{+}}P_{ij}^{-}. This gives Eq. ( 10 ).

Using P i + + ∑ j = 1 N − P i ​ j − = 1 P_{i}^{+}+\sum_{j=1}^{N^{-}}P_{ij}^{-}=1 , we obtain ∑ j = 1 N − ∂ 𝒥 ^ NCE ​ ( q ) ∂ s j − \displaystyle\sum_{j=1}^{N^{-}}\frac{\partial\widehat{\mathcal{J}}_{\rm NCE}(q)}{\partial s_{j}^{-}} = − 1 N + ∑ i = 1 N + ∑ j = 1 N − P i ​ j − = − 1 N + ∑ i = 1 N + ( 1 − P i + ) . \displaystyle=-\frac{1}{N^{+}}\sum_{i=1}^{N^{+}}\sum_{j=1}^{N^{-}}P_{ij}^{-}=-\frac{1}{N^{+}}\sum_{i=1}^{N^{+}}(1-P_{i}^{+}). Moreover, ∑ i = 1 N + ∂ 𝒥 ^ NCE ​ ( q ) ∂ s i + + ∑ j = 1 N − ∂ 𝒥 ^ NCE ​ ( q ) ∂ s j − \displaystyle\sum_{i=1}^{N^{+}}\frac{\partial\widehat{\mathcal{J}}_{\rm NCE}(q)}{\partial s_{i}^{+}}+\sum_{j=1}^{N^{-}}\frac{\partial\widehat{\mathcal{J}}_{\rm NCE}(q)}{\partial s_{j}^{-}} = 1 N + ​ ∑ i = 1 N + ( 1 − P i + ) − 1 N + ​ ∑ i = 1 N + ( 1 − P i + ) = 0 . \displaystyle=\frac{1}{N^{+}}\sum_{i=1}^{N^{+}}(1-P_{i}^{+})-\frac{1}{N^{+}}\sum_{i=1}^{N^{+}}(1-P_{i}^{+})=0. This gives Eq. ( 11 ).

Finally, for any two negative rollouts o j − o_{j}^{-} and o k − o_{k}^{-} , P i ​ j − \displaystyle P_{ij}^{-} = exp ⁡ ( s j − / τ ) Z i = exp ⁡ ( s j − − s k − τ ) ​ exp ⁡ ( s k − / τ ) Z i = exp ⁡ ( s j − − s k − τ ) ​ P i ​ k − . \displaystyle=\frac{\exp(s_{j}^{-}/\tau)}{Z_{i}}=\exp\left(\frac{s_{j}^{-}-s_{k}^{-}}{\tau}\right)\frac{\exp(s_{k}^{-}/\tau)}{Z_{i}}=\exp\left(\frac{s_{j}^{-}-s_{k}^{-}}{\tau}\right)P_{ik}^{-}. Summing over i i yields ∑ i = 1 N + P i ​ j − = exp ⁡ ( s j − − s k − τ ) ​ ∑ i = 1 N + P i ​ k − . \sum_{i=1}^{N^{+}}P_{ij}^{-}=\exp\left(\frac{s_{j}^{-}-s_{k}^{-}}{\tau}\right)\sum_{i=1}^{N^{+}}P_{ik}^{-}. Combining this identity with Eq. ( 10 ) gives | ∂ 𝒥 ^ NCE ​ ( q ) / ∂ s j − | | ∂ 𝒥 ^ NCE ​ ( q ) / ∂ s k − | = ∑ i = 1 N + P i ​ j − ∑ i = 1 N + P i ​ k − = exp ⁡ ( s j − − s k − τ ) . \frac{\left|\partial\widehat{\mathcal{J}}_{\rm NCE}(q)/\partial s_{j}^{-}\right|}{\left|\partial\widehat{\mathcal{J}}_{\rm NCE}(q)/\partial s_{k}^{-}\right|}=\frac{\sum_{i=1}^{N^{+}}P_{ij}^{-}}{\sum_{i=1}^{N^{+}}P_{ik}^{-}}=\exp\left(\frac{s_{j}^{-}-s_{k}^{-}}{\tau}\right). This gives Eq. ( 12 ). ∎

## Appendix B Detailed Related Work

#### RLVR for LLM reasoning.

GRPO ( 11 ) circumvents the reliance on a parameterized value model by estimating group-relative advantages directly from outcome-level rewards, maintaining robust performance across diverse reasoning tasks. Building on this paradigm, subsequent efforts have advanced GRPO toward better performance, more stable training, and more efficient reasoning. Dr. GRPO ( 20 ) removes the standard deviation in advantage estimation to mitigate difficulty bias, and adjusts loss allocation to reduce length bias. DAPO ( 35 ) introduces dynamic sampling to reduce high-variance gradient updates, and employs length-based reward shaping for finer-grained length control. GMPO ( 39 ) addresses outlier effects in asynchronous training by replacing token-level clipping with sequence-level geometric-mean clipping. CISPO ( 4 ) introduces soft clipping to preserve learning signals discarded by hard clipping, while SAPO ( 9 ) applies asymmetric, temperature-controlled clipping for finer-grained and stable updates. HAPO ( 15 ) and ShorterBetter ( 34 ) promote concise reasoning by selecting shorter trajectories from either historical or current generations. While DisCO ( 18 ) reinterprets GRPO through a discriminative lens under binary rewards and explores alternative scoring functions, it does not address the score-insensitive credit assignment induced by the GRPO objective. In contrast, ConSPO optimizes a group-wise InfoNCE-style objective for online RLVR, using likelihood-aligned sequence scores and a curriculum-scheduled margin to enable contrast-sensitive credit assignment within rollout groups.

#### Contrastive Objectives for Policy Alignment.