GRPO-VPS: Enhancing Group Relative Policy Optimization with Verifiable Process Supervision for Effective Reasoning — part 2/3

Cleaned-text code points [15692, 31684). Bibliography omitted here; source.md includes it.

Models and baselines. We conduct experiments on two model families, including Qwen2.5-Math-1.5B, Qwen2.5-Math-7B ( Yang et al., 2024 ) and Gemma-2-2B-it ( Team, 2024a ) . To ensure fair comparison, we include a comprehensive set of baselines categorized by their use of outcome-level vs. process-level supervision: • Outcome Supervision Only. This category includes methods that rely solely on final answer correctness for reward assignment. We consider GRPO and its recent variants, DrGRPO ( Liu et al., 2025 ) and GSPO ( Zheng et al., 2025 ) , which enhance group-wise comparison or propagate advantages with entropy-based mechanisms. We also include the BASE models without RL fine-tuning for reference.

• With Process Supervision. This group covers methods that incorporate intermediate supervision beyond outcome-level rewards. We evaluate S-GRPO ( Dai et al., 2025 ) , which relies on Monte Carlo rollouts with forced early stops to construct sub-trajectories, and assigns segment-level rewards based on their predicted outcomes. We also compare against PRIME-style reward modeling, represented by the public Eurus-2-7B-PRIME ( Cui et al., 2025 ; Yuan et al., 2024 ) , and a controlled variant where we fine-tune Qwen2.5-Math-1.5B and 7B with Skywork-o1-prm using GRPO. These baselines provide strong comparisons for evaluating the effectiveness of verifiable process supervision in our method.

Training setup. In line with prior work ( Liu et al., 2025 ) , we use MATH ( Hendrycks et al., 2021 ) , which contains 7,500 problems. We train the models using the verl framework ( Sheng et al., 2024 ) . We sample 8 rollouts per prompt, with a temperature of 1.0 and the maximum response length of 3,072 tokens. The batch size is set to 512, the mini-batch size to 128, and the learning rate to 1 × 10 − 6 1\times 10^{-6} . The training is conducted on a single node with 8 × H800 GPUs. More hyperparameter settings can be found in Appendix A.1 .

Evaluation setup. We evaluate on four widely used math reasoning benchmarks, including the test sets of MATH, AIME 2024 ( MAA, 2024 ) , AMC23 ( MAA, 2023 ) and OlympiadBench ( He et al., 2024a ) . We set temperature to 1.0, top p p set to 1, and maximum output length set to 3,072 tokens for inference. Due to the high variance of the outputs from reasoning models, we report the average Pass@1 over 4 runs. To ensure accurate evaluation, we utilize Math-Verify 1 1 1 https://github.com/huggingface/Math-Verify to check for answer equivalence.

### 4.2 Main Results

GRPO-VPS improves mathematical reasoning performance. As shown in Table 1 , our method achieves the highest average accuracy. On Qwen2.5-Math-1.5B, it yields an average gain of 27.7 points. On Qwen2.5-Math-7B, the gain reaches +31.0 points overall. Meanwhile, the average output length is reduced by 35.9% and 34.0% on the 1.5B and 7B models, respectively. Similar trends are observed on Gemma-2-2B-it, where our method improves average accuracy from 6.9 to 11.7 while substantially reducing output length compared to GRPO.

Comparison with outcome supervised RL. Compared to GRPO and its recent variants, GRPO-VPS achieves the highest overall accuracy across all benchmarks while substantially shortening the reasoning length. On Qwen2.5-Math-1.5B, it improves Pass@1 by more than 3% over GRPO and sustains a comparable margin on the 7B model, with an average reduction of 10–11% in output length. GRPO-VPS achieves larger gains in accuracy and produces shorter outputs. These results suggest that our method improves both effectiveness and efficiency of policy updates, leading to more focused reasoning traces. On Gemma-2-2B-it, our method further reduces the output length compared to GRPO while achieving higher accuracy. Although the Gemma base model exhibits shorter responses, training with reinforcement learning unlocks its long-chain reasoning behavior, as further evidenced in Appendix A.4.1 .

Comparison with model-based process supervision. Compared to S-GRPO, which uses multi-rollout early-exit probing to assign segment-level rewards, our method achieves higher accuracy and shorter outputs. To ensure a fair comparison, we adopt the same GRPO training pipeline to finetune both our method and Skywork-o1-prm. Under this controlled setup, GRPO-VPS consistently outperforms the Skywork baseline across both 1.5B and 7B model scales, achieving higher accuracy while generating more concise reasoning traces. We further compare our method with Eurus-2-7B-PRIME, a model trained using PRIME-style reward modeling. Notably, Eurus performs poorly on AIME24 with 15.0%, substantially lower than the 31.7% achieved by GRPO-VPS. This discrepancy reveals a lack of generalization in PRM-based methods, which tend to overfit to familiar patterns seen during training. In contrast, our process supervision, grounded in the model’s own belief dynamics, offers more robust and consistent improvements across tasks, while reducing reasoning length by up to 34.0%.

### 4.3 Ablation Study

Effect of segment granularity. To investigate the impact of segment granularity on the efficiency and performance of our adaptive strategy, we evaluate the impact of segment granularity by varying the average number of points n n per segment. As shown in Figure 2 (a), n = 4 n=4 achieves the optimal trade-off between training efficiency and performance under the same wall-clock time. While finer-grained segmentation ( n = 2 n=2 ) provides more precise local adjustments, it incurs prohibitive computational overhead that slows convergence. This confirms that our segment-level design effectively strikes a balance between computational economy and optimization precision, avoiding the excessive costs associated with fine-grained estimation while ensuring superior learning dynamics.

Comparison of Different segment strategies. We compare our adaptive segmentation strategy with a naive fixed token-count partition baseline, where each response is evenly divided into a fixed number of segments. For the fixed baseline, we set the number of segments to 6, exceeding the effective segmentation budget of our adaptive method. As shown in Figure 2 (b), the adaptive strategy converges faster, achieves higher accuracy, and maintains shorter responses throughout training. These results demonstrate that placing segment boundaries based on informative decision points is more effective than uniform partitioning, validating the design of our adaptive segmentation strategy.

Effect of outcome supervision signal. As shown in Table 2 , removing the outcome reward and relying solely on segment-level verifiable supervision results in degradation compared to the full method, confirming its essential role in guiding the model toward globally correct reasoning. While segment-level supervision alone provides fine-grained feedback, it lacks a reliable global anchor. Combining both signals yields the best performance, indicating their complementarity in stabilizing training and improving reasoning coherence.

### 4.4 Understanding How VPS Works

Quality analysis for segment-wise process signal. A core premise of our approach is that the segment-wise process signal ( Δ ​ C k \Delta C_{k} ) provides a reliable proxy for the correctness of intermediate reasoning steps. Unlike outcome-level signals, which uniformly credit or penalize every token in a trajectory, Δ ​ C k \Delta C_{k} directly reflects how each step changes the model’s belief in the correct answer.

To quantitatively validate this hypothesis, we align our progress signal with segment-level human annotations from the PRM800K dataset ( Lightman et al., 2023 ) ,which contains 800K reasoning steps labeled as -1 (incorrect/harmful), 0(neutral/uninformative), or +1 (correct/contributive). For evaluation, we randomly sample 100 held-out questions, each paired with six diverse model responses. For each reasoning segment, we discard neutral steps with label 0 and discretize Δ ​ C k \Delta C_{k} into predicted class labels.

Table 3 reports the precision, recall, and F1-score across model scales from 0.5B to 32B parameters ( Team, 2024b ) . We observe that even small models produce progress signals that meaningfully discriminate good from bad reasoning steps, with F1-scores exceeding 0.75 across the board. Larger models further improve both precision and recall, demonstrating that Δ ​ C k \Delta C_{k} scales naturally with model quality. These results confirm that our segment-wise progress signal serves as a lightweight yet effective indicator of reasoning quality, complementing trajectory-level outcome signals. Additional qualitative examples illustrating this alignment with human judgment are provided in the Appendix A.5 .

Sample efficiency and optimization stability. Recent studies prove dense reward signals can improve the sample efficiency and optimization stability of reinforcement learning systems by providing more frequent and informative feedback during training ( Setlur et al., 2024 ; Chan et al., 2024 ) . Our method introduces verifiable, model-free process supervision to construct dense, segment-wise signals that guide the optimization process more precisely than trajectory-level binary rewards alone. As shown in Figure 3 , our method achieves faster convergence and more stable gradient updates compared to standard GRPO.

### 4.5 General Reasoning Benchmarks

To verify the generalization capabilities of our method beyond specific domain tasks, we extended our evaluation to a suite of general reasoning benchmarks. Following the experimental setup in RLPR ( Yu et al., 2025b ) , we utilized the WebInstruct dataset ( Ma et al., 2025 ) for training, this dataset is characterized by a diverse semantic distribution, covering a wide range of disciplines including Physics, Mathematics Business, and Economics, thereby requiring the model to possess robust multi-domain reasoning abilities. We employed Qwen3-1.7B ( Team, 2025 ) as the backbone model and compared our proposed method against the Base model and the GRPO baseline. The evaluation was conducted on four challenging benchmarks: GPQA ( Rein et al., 2024 ) , MMLUPro ( Wang et al., 2024 ) , TheoremQA ( Chen et al., ) , and the test split of WebInstruct. As shown in Figure 4 , GRPO-VPS consistently outperforms baselines across all tasks. On GPQA and TheoremQA, it achieves 37.8% and 37.3% accuracy, surpassing GRPO (by 1.6% and 2.4%) and significantly beating the Base model (by 20.0% and 12.8%). Similarly, on MMLUPro and WebInstruct, our method reaches 54.4% and 73.9%, exceeding GRPO by 1.7% and 3.6%, respectively. Furthermore, these gains are achieved with greater efficiency. Our method reduces the average generation length by nearly 50% compared to the Base model, surpassing the GRPO baseline in conciseness. These results confirm that our method effectively enhances robust, multi-domain reasoning. Additional results and training details can be found in Appendix A.4.2 .

## 5 Conclusion

We present GRPO-VPS, a model-free and verifier-free method that augments GRPO with segment-level credit assignment derived from conditional answer probabilities. Our method generates dense and interpretable supervision signals aligned with the model’s internal decision flow, enabling more efficient and targeted policy optimization. Experiments on four math reasoning benchmarks show that our method consistently improves both accuracy and reasoning conciseness over strong RL baselines and segment-aware methods, without relying on auxiliary models or costly rollouts. Furthermore, extensive evaluations on general reasoning benchmarks confirm the method’s strong generalization capabilities, demonstrating consistent gains in robust, multi-domain reasoning tasks. These findings underscore the potential of verifier-free, confidence-driven rewards as a scalable direction for future alignment and reasoning optimization in large language models.

## Appendix A Appendix

### A.1 Experimental Settings

All hyperparameter settings are listed in Table 4 , our experiments are performed on 8 × H100 GPUs.

### A.2 Prompt Templates

For math reasoning tasks, we adopt the prompt templates for Qwen Math families ( Yang et al., 2024 ) and Gemma ( Team, 2024a ) .

For general reasoning tasks, we adopt the prompt templates for Qwen3 ( Team, 2025 ) model.

### A.3 Disclosure of LLM usage.

This paper benefited from language editing and phrasing suggestions provided by ChatGPT (OpenAI), which was used solely for grammar correction and clarity improvement. No LLM was used for generating research ideas, experimental design, data analysis, or writing substantive technical content.

### A.4 Extended Empirical Results

#### A.4.1 Response Length Dynamics across Model Families

We observe opposite response-length dynamics between Gemma and Qwen Math models under reinforcement learning, as illustrated in Figure 5 . Gemma base model exhibits extremely short responses at initialization, due to its instruction-tuned alignment, which explicitly suppresses verbosity and favors concise, direct answers. As training progresses, both methods gradually increase the response length, as longer reasoning trajectories are consistently associated with higher success probability and thus receive positive reinforcement.

In contrast, Qwen base models initially tend to produce more redundant or repetitive content during training process, the models learn to compress these redundant reasoning steps, leading to significantly shorter and more focused outputs.

#### A.4.2 Additional Analysis on General Reasoning Benchmarks

Fine-grained results on MMLU-Pro. We provide detailed results on the MMLU-Pro test set by reporting accuracy across individual subject domains. Following prior work, we adopt abbreviated domain names with the full nomenclature as follows: Math (Mathematics), Bio (Biology), Econ (Economics), Chem (Chemistry), Bus (Business), CS (Computer Science), Phys (Physics), Psy (Psychology), Eng (Engineering), Health (Health), Other (Other), Phil (Philosophy), Hist (History), and Law (Law). Table 5 reports per-domain accuracy together with the average response length. Compared to both the Base model and GRPO, our method achieves the highest average accuracy while producing shorter responses.

Training and evaluation performance for general reasoning. Figure 6 further illustrates the training entropy loss curves and test accuracy on TheoremQA. Compared to GRPO, our method exhibits a consistently lower entropy loss throughout training, indicating more stable and confident policy updates.

#### A.4.3 Sensitivity Analysis on α \alpha

We further analyze the sensitivity of the weighting factor α \alpha , which balances the trajectory-level outcome advantage and the segment-level process supervision in Eq. ( 6 ). Specifically, α \alpha controls the relative contribution of the segment-wise progress signal Δ ​ C k \Delta C_{k} to the overall hybrid advantage.

We conduct a sensitivity study by varying α \alpha over a wide range while keeping all other training settings fixed. As shown in Table 6 , our method achieves the best performance at α \alpha = 1.2. Moreover, performance is relatively insensitive to α \alpha within the range [0.8, 1.4].

### A.5 Qualitative Analysis of Segment-wise Process Signals.

To qualitatively evaluate the effectiveness of our progress signal, we present representative cases where the segment-wise signal Δ ​ C k \Delta C_{k} successfully distinguishes contributive from harmful reasoning steps. As shown in Figure 8 and 7 , positive increments in Δ ​ C k \Delta C_{k} correspond to steps that strengthen the model’s belief in the correct answer, while negative increments highlight misleading or incorrect reasoning. These examples demonstrate that our process supervision can reliably identify the strengths and weaknesses within a model’s reasoning trajectory, providing interpretable and fine-grained feedback throughout the generation process.