Revisiting Reinforcement Learning with Verifiable Rewards from a Contrastive Perspective — part 4/4

Cleaned-text code points [47477, 52382). Bibliography omitted here; source.md includes it.

Contrastive learning trains models by distinguishing positive samples from negative distractors, with InfoNCE-style objectives serving as a standard formulation ( 24 ) . Foundational methods improve contrastive learning by enlarging the negative set, for example, through large-batch sampling in SimCLR ( 6 ) or momentum queues in MoCo ( 13 ) . Inspired by contrastive learning principles, recent policy alignment methods formulate preference optimization as a task of discriminating between preferred and dispreferred responses. For example, DPO ( 25 ) and CPO ( 31 ) optimize policy likelihoods over preference pairs without explicit reward modeling, while SimPO ( 22 ) further uses length-normalized sequence log-probabilities as an implicit ranking score. Unlike these offline preference optimization methods, ConSPO transfers the contrastive principle to online RLVR, where positive-negative comparisons are constructed from verifiable rewards on policy-generated rollouts rather than from static pairwise preference data.

## Appendix C Detailed Experimental Settings

### C.1 Benchmarks

We evaluate the mathematical reasoning ability of the trained models on seven benchmarks that are commonly used in recent RLVR studies ( 28 ; 32 ; 36 ; 38 ) .

• AIME24–26 — Problems from the 2024, 2025, and 2026 American Invitational Mathematics Examination. AIME consists of challenging high-school competition problems with integer-valued answers, requiring precise multi-step reasoning rather than multiple-choice selection.

• HMMT25 — Problems from the February 2025 Harvard–MIT Mathematics Tournament. The benchmark covers competition-level problems in algebra and number theory, combinatorics, and geometry, as well as team-based rounds.

• MATH500 — A 500-problem evaluation subset of the MATH benchmark. It contains competition-style problems across seven subjects, including algebra, geometry, counting and probability, number theory, and precalculus.

• AMC — Problems from the American Mathematics Competitions, covering a broad range of high-school mathematics topics. Although AMC is originally multiple-choice, the problems still require careful reasoning and symbolic manipulation.

• OlympiadBench — An Olympiad-level benchmark containing challenging mathematics and physics problems. It evaluates advanced scientific reasoning on problems collected from Olympiad-style competitions and related examinations.

Following prior work ( 21 ) , we report avg@32 on AIME, HMMT, and AMC to reduce the variance caused by their relatively small test sizes, and report pass@1 on the remaining benchmarks.

### C.2 Implementation Details

The training and evaluation configurations for the R1-Distill models are summarized in Tables 7 and 8 , respectively. For the experiments on Qwen3-4B-Base and Llama-3.2-3B-Instruct, we use the same settings as the R1-Distill models, except that the maximum response length is set to 4096 during both training and evaluation. In addition, Qwen3-4B-Base is trained for 2 epochs due to its faster convergence, while the other models are trained for 4 epochs.

## Appendix D Additional Experimental Results

To further evaluate the scalability of ConSPO, we conduct additional experiments on DeepSeek-R1-Distill-Qwen-32B. We use the same training data and hyperparameter settings as in the main experiments, and train both GRPO and ConSPO for two epochs. As shown in Table 9 , ConSPO outperforms GRPO across all seven benchmarks, improving the average score from 60.2 to 63.0. These results demonstrate that the effectiveness of ConSPO extends to substantially larger model scales.

## Appendix E Prompt Template

We use a unified prompt template for both rollout generation and evaluation across all methods. As shown in Figure 4 , the prompt asks the model to solve the problem step by step and place the final answer within \boxed{} , which facilitates answer extraction and rule-based verification.

## Instructions for reporting errors

We are continuing to improve HTML versions of papers, and your feedback helps enhance accessibility and mobile support. To report errors in the HTML that will help us improve conversion and rendering, choose any of the methods listed below:

Click the "Report Issue" ( ) button, located in the page header.

Tip: You can select the relevant text first, to include it in your report.

Our team has already identified the following issues . We appreciate your time reviewing and reporting rendering errors we may not have found yet. Your efforts will help us improve the HTML versions for all readers, because disability should not be a barrier to accessing research. Thank you for your continued support in championing open access for all.

Have a free development cycle? Help support accessibility at arXiv! Our collaborators at LaTeXML maintain a list of packages that need conversion , and welcome developer contributions .