GRPO-VPS: Enhancing Group Relative Policy Optimization with Verifiable Process Supervision for Effective Reasoning — part 1/3

Cleaned-text code points [0, 15692). Bibliography omitted here; source.md includes it.

# GRPO-VPS: Enhancing Group Relative Policy Optimization with Verifiable Process Supervision for Effective Reasoning

###### Abstract

Reinforcement Learning with Verifiable Rewards (RLVR) has advanced the reasoning capabilities of Large Language Models (LLMs) by leveraging direct outcome verification instead of learned reward models. Building on this paradigm, Group Relative Policy Optimization (GRPO) eliminates the need for critic models but suffers from indiscriminate credit assignment for intermediate steps, which limits its ability to identify effective reasoning strategies and incurs overthinking. In this work, we introduce a model-free and verifiable process supervision via probing the model’s belief in the correct answer throughout its reasoning trajectory. By segmenting the generation into discrete steps and tracking the conditional probability of the correct answer appended at each segment boundary, we efficiently compute interpretable segment-wise progress measurements to refine GRPO’s trajectory-level feedback. This approach enables more targeted and sample-efficient policy updates, while avoiding the need for intermediate supervision derived from costly Monte Carlo rollouts or auxiliary models. Experiments on mathematical and general‑domain benchmarks show consistent gains over GRPO across diverse models: up to 2.6‑point accuracy improvements and 13.7% reasoning‑length reductions on math tasks, and up to 2.4 points and 4% on general‑domain tasks, demonstrating strong generalization.

## 1 Introduction

Advanced by Reinforcement Learning with Verifiable Rewards (RLVR) ( Shao et al., 2024 ; Yu et al., 2025a ) , Large Language Models (LLMs) have demonstrated remarkable capabilities in complex reasoning tasks, ranging from mathematical problem solving ( OpenAI, 2024 ; Team et al., 2025 ; Shao et al., 2024 ) to multi-hop question answering ( Huang et al., 2025 ; Song et al., 2025 ) . The success of RLVR is largely attributed to computing rewards via direct outcome verification, rather than relying on reward models that complicate the training pipeline and are prone to reward hacking ( Yu et al., 2025a ) . In a similar vein, Group Relative Policy Optimization (GRPO) ( Shao et al., 2024 ) eliminates critic models for token-level advantage estimation, instead uniformly propagating trajectory-level advantages to intermediate steps. While this simplification avoids the challenges of training a critic model and reduces associated overhead, indiscriminate credit assignment hinders sample efficiency and limits the policy model’s ability to learn effective reasoning strategies ( Qu et al., 2025 ) .

To address this limitation, we explore enhancing GRPO with model-free, verifiable process supervision derived from the annotated final answer. Our key insight is that the contribution of intermediate reasoning steps can be probed by the probability increment of the reference answer appended at corresponding breakpoints. This is supported by observations in Figure 1 : (1) at the macro level, the average probed probability increases as reasoning progresses, with a more pronounced trend for trajectories that ultimately reach the correct answer; and (2) at the micro level, reasoning segments that reduce the probed probability tend to be of low quality. Based on these observations, our method leverages the model’s own reasoning trace to generate localized supervision signals, enabling more targeted and effective policy updates. Specifically, we segment the model’s response into discrete reasoning segments and strategically concatenate the correct final answer at each segment boundary. By extracting the model’s conditional probability of the correct answer at these positions, we obtain a proxy for its evolving belief state. The differences in these probabilities between adjacent segments serve as segment-wise supervision signals, complementing trajectory-level advantages and quantifying the contribution of each reasoning segment toward the final outcome.

This approach offers two key benefits: (1) it provides dense, interpretable feedback aligned with the model’s internal decision flow and (2) it avoids reliance on auxiliary models ( Schulman et al., 2017 ; Zha et al., 2025 ; Cui et al., 2025 ; He et al., 2024b ) or Monte Carlo rollouts ( Qu et al., 2025 ; Dai et al., 2025 ) , ensuring high efficiency and scalability and adhering to the design principles established by RLVR and GRPO. Through this fine-grained supervision mechanism, we aim to enhance the sample efficiency of RL training, paving the way for learning more effective and efficient reasoning behaviors.

Our experiments show substantial gains across four math reasoning benchmarks. Compared to GRPO, our method achieves up to +2.6 points Pass@1 on Qwen2.5-Math-1.5B and +1.1 point on Qwen2.5-Math-7B, while concurrently reducing reasoning length by 11.0% to 13.7%. It also consistently outperforms the GRPO variant ( Dai et al., 2025 ) that relies on costly Monte Carlo rollouts for segment-wise advantage estimation and auxiliary models ( Cui et al., 2025 ; He et al., 2024b ; Schulman et al., 2017 ) . Furthermore, evaluation on four general-domain reasoning benchmarks confirms strong generalization, with gains of 1.8 points on MMLUPro and 2.4 points on TheoremQA. These results highlight the effectiveness and scalability of our verifiable process supervision in delivering more accurate and concise reasoning.

In summary, our main contributions are: • We identify and empirically validate that a model’s evolving belief in the correct answer can serve as a model-free, interpretable signal for reasoning quality of intermediate steps. This enables fine-grained supervision without auxiliary models or Monte Carlo rollouts.

• We propose GRPO-VPS, a simple yet effective approach to enhance GRPO with granular, segment-wise process supervision, avoiding indiscriminate credit assignment and improving sample efficiency.

• Our empirical results show that the method achieves strong performance on challenging math reasoning tasks, demonstrating that our method enhances both reasoning effectiveness and efficiency in comparison with GRPO and its variants.

## 2 Related Work

Group Relative Policy Optimization (GRPO) . Reinforcement Learning with Verifiable Rewards (RLVR) has become a prominent paradigm for fine-tuning LLMs, using definitive signals from rule-based verifiers to circumvent the need for costly and potentially biased reward models ( Shao et al., 2024 ; Yu et al., 2025a ) . Within this paradigm, GRPO ( Shao et al., 2024 ) offers a lightweight and efficient alternative to critic-based algorithms like PPO ( Schulman et al., 2017 ) . By comparing final outcomes across a group of sampled trajectories, GRPO eliminates the need for a separate value network. However, this simplification comes at the cost of indiscriminate credit assignment: a single, trajectory-level reward is uniformly propagated to all intermediate tokens. This can inadvertently reinforce spurious reasoning steps in a successful trajectory or penalize promising partial logic in a failed one. Our work addresses this limitation by introducing a fine-grained process supervision mechanism, enhancing its credit assignment capabilities without sacrificing its lightweight nature.

Process supervision for reasoning . Recent work has explored injecting fine-grained supervision into the reasoning process to better guide long-form generation. These efforts can be broadly categorized into model-based and model-free approaches. Model-based supervision utilizes an auxiliary model to provide fine-grained feedback. Critic-based methods, often using PPO, train a value network to estimate the expected return from intermediate states ( Yue et al., 2025 ; Kazemnejad et al., 2024 ) . However, in long-horizon reasoning tasks, the critic’s signal can diminish or become unreliable due to the long delay in receiving the final outcome reward ( Shao et al., 2024 ; Yue et al., 2025 ) . Process Reward Models (PRMs) ( Lightman et al., 2023 ; Wang et al., 2023 ) offer an alternative but are typically trained offline, making them vulnerable to reward hacking and distributional shift. These approaches introduce significant system complexity, requiring an extra model to be trained, maintained, and served alongside the policy. Model-free process supervision aims to provide granular feedback without auxiliary models. Recent works have made progress in this direction. For instance, S-GRPO ( Dai et al., 2025 ) introduces a ”serial group” objective with decaying rewards to encourage earlier, more efficient reasoning. MRT ( Qu et al., 2025 ) frames the problem as meta reinforcement learning and computes a dense “progress” reward based on the change in the likelihood of eventual success. Their reward estimation can require complex, rollout-based procedures or multiple generation branches from intermediate states, which compromises training efficiency. In contrast, our method simplifies the process supervision workflow by deriving a high-quality signal from the known ground-truth answer, requiring only a single forward pass per generated trajectory. This makes our approach more efficient while still providing the benefits of fine-grained, verifiable process feedback.

## 3 Methodology: Process Supervisions from Verifiable Outcomes

LLMs trained under GRPO still suffer from indiscriminate credit assignment, where sparse outcome-based rewards fail to guide intermediate reasoning steps. To address this, we present a verifiable process supervision framework for enhancing GRPO with fine-grained credit assignment. We first introduce a segmentation strategy that uses token-level entropy to identify high-uncertainty transitions and partition trajectories into semantically meaningful reasoning steps (Section 3.1 ). We then introduce segment-wise progress estimation to quantify the contribution of each reasoning segment based on changes in model confidence (Section 3.2 ). Finally, we incorporate this localized feedback into GRPO’s token-level updates, forming a hybrid advantage that fuses outcome-based and process-level signals (Section 3.3 ).

### 3.1 Reasoning Process Segmentation

Recent studies have revealed that performance gains in RLVR are primarily driven by critical decision points characterized by high token-level uncertainty ( Yang et al., 2025 ; Wang et al., 2025 ) . Inspired by SPO ( Guo et al., 2025 ) , we adopt an Adaptive Entropy-based Cutpoint Partition strategy, leveraging token-level entropy to robustly identify reasoning ”junctions” where the model’s trajectory is likely to diverge.

Formally, given a response o o of length T T , We identify a set of candidate cutpoints 𝒰 ⊆ { 1 , … , T } \mathcal{U}\subseteq\{1,\dots,T\} by selecting tokens whose entropy exceeds an adaptive threshold: 𝒰 = { t ∣ e t i ≥ τ } , \mathcal{U}=\{t\mid e_{t}^{i}\geq\tau\}, (1) where τ \tau is determined from the entropy distribution of o o (e.g. via a perentile-based rule). To partition o o into M M reasoning segments ( z 1 , … , z M ) (z_{1},\dots,z_{M}) , we choose boundary indices { t 1 , … , t M + 1 } \{t_{1},\dots,t_{M+1}\} with t 1 = 1 t_{1}=1 and t M + 1 = T + 1 t_{M+1}=T+1 , such that the number of cutpoints in each segment is approximately balanced: | 𝒰 ∩ [ t m , t m + 1 ) | ≈ | 𝒰 | M , ∀ m ∈ { 1 , … , M } . |\mathcal{U}\cap[t_{m},t_{m+1})|\approx\frac{|\mathcal{U}|}{M},\quad\forall m\in\{1,\dots,M\}. (2) This heuristic ensures that each segment contains a comparable number of high-entropy positions, yielding a balanced and semantically meaningful segmentation of the reasoning trajectory. Our experiments demonstrate that this adaptive strategy yields superior performance compared to fixed-token partition (see Section 4.3 ).

### 3.2 Progress as Process Supervision

Based on the reasoning process segmentation, we propose to leverage segment-wise progress as a form of process supervision to address the indiscriminate credit assignment of GRPO. This formulation provides a dense, model-free, and scalable supervision signal that quantifies the incremental contribution of each reasoning segment toward the correct final answer.

Given an input prompt x x and a trajectory o o generated by the policy π θ \pi_{\theta} , we compute a segment-wise confidence score C ⁡ ( z ≤ k ) C(z_{\leq k}) representing the model’s conditional probability of the target answer y ∗ y^{*} after generating the first k k reasoning steps:

C ⁡ ( z ≤ k ) = π θ ​ ( y ∗ ∣ x , z ≤ k ) C(z_{\leq k})=\pi_{\theta}(y^{*}\mid x,z_{\leq k}) (3)

where x x is the input question and z ≤ k = ( z 1 , … , z k ) z_{\leq k}=(z_{1},\ldots,z_{k}) denotes partial reasoning trace up to segment k k . The initial value, before any reasoning, is C 0 = P ⁡ ( y ∗ ∣ x ) C_{0}=P(y^{*}\mid x) .

To quantify the contribution of each reasoning segment, we define a segment-wise progress score, denoted as Δ ​ C k \Delta C_{k} , is then computed as the change in this confidence score, effectively isolating the contribution of that specific step: Δ ​ C k \displaystyle\Delta C_{k} = C ⁡ ( z ≤ k ) − C ⁡ ( z ≤ k − 1 ) \displaystyle=C(z_{\leq k})-C(z_{\leq k-1}) = π θ ​ ( y ∗ ∣ x , z ≤ k ) − π θ ​ ( y ∗ ∣ x , z ≤ k − 1 ) \displaystyle=\pi_{\theta}(y^{*}\mid x,z_{\leq k})-\pi_{\theta}(y^{*}\mid x,z_{\leq k-1}) where Δ ​ C k ∈ [ − 1 , 1 ] \Delta C_{k}\in[-1,1] due to the probabilistic range of confidence scores. This results in a vector Δ ​ C = [ Δ ​ C 1 , … , Δ ​ C m ] \Delta C=[\Delta C_{1},\ldots,\Delta C_{m}] of segment-level supervision signals for each trajectory. It reflects how much each segment improves (or worsens) the model’s belief in the final answer.

### 3.3 GRPO with Process Supervision

We design a hybrid advantage signal that fuses sparse outcome-level feedback with dense, segment-level process supervision. For each prompt, we sample a group of G G trajectories { o 1 , o 2 , … , o G } \{o^{1},o^{2},\ldots,o^{G}\} from the policy π θ \pi_{\theta} . Each trajectory o i = ( z 1 i , … , z m i , y i ) o^{i}=(z_{1}^{i},\ldots,z_{m}^{i},y^{i}) with binary correctness label r i ∈ { 0 , 1 } r^{i}\in\{0,1\} , we compute the group-relative advantage: A i = r i − 1 G ​ ∑ j = 1 G r j , A^{i}=r^{i}-\frac{1}{G}\sum_{j=1}^{G}r^{j}, (5) where G G is the number of responses sampled for the same prompt. To complement this global signal, we inject a localized feedback term Δ ​ C k \Delta C_{k} , which quantifies the incremental gain in the model’s belief in the correct answer after each reasoning segment z k i z_{k}^{i} , as defined in Section 3.2 . The final hybrid advantage at step k k is: A ~ k i = A i ⏟ Outcome + α ⋅ Δ ​ C k ⏟ Process , \tilde{A}_{k}^{i}=\underbrace{A^{i}}_{\text{Outcome}}+\underbrace{\alpha\cdot\Delta C_{k}}_{\text{Process}}, (6) where α \alpha is a weighting factor balancing the two components. We empirically set α = 1.2 \alpha=1.2 and found it work well. Sensitivity to α \alpha can be found in Appendix A.4.3 .

We then define the final on-policy gradient estimator as:

∇ θ J ​ ( θ ) = 1 G ​ ∑ i = 1 G ∑ k = 1 M ( A i + α ⋅ Δ ​ C k ) ⋅ ∇ θ ​ log ​ π θ ​ ( z k i ∣ x , z < k i ) \nabla_{\theta}J(\theta)=\frac{1}{G}\sum_{i=1}^{G}\sum_{k=1}^{M}\left(A^{i}+\alpha\cdot\Delta C_{k}\right)\cdot\nabla_{\theta}\log\pi_{\theta}(z_{k}^{i}\mid x,z_{<k}^{i}) (7) where the total advantage combines two signals: A i A^{i} provides sparse, trajectory-level feedback based on the final outcome, while α ⋅ Δ ​ C k \alpha\cdot\Delta C_{k} injects dense, segment-level guidance reflecting the progress toward the correct answer. The full algorithm of GRPO-VPS is shown in Algorithm 1 .

## 4 Experiment

### 4.1 Setup