Learning from Own Solutions: Self-Conditioned Credit Assignment for Reinforcement Learning with Verifiable Rewards — part 2/4
Cleaned-text code points [15761, 31676). Bibliography omitted here; source.md includes it.
This shift, quantified as token-level KL divergence, identifies which tokens depend on access to a verified solution. Tokens with small KL make the same prediction regardless of whether a verified solution is available. The token’s choice is unrelated to the rollout’s outcome, so applying sequence-level credit here misattributes success or failure. Tokens with large KL depend on verified solution, so the reward signal is informative. SC-GRPO uses this KL purely as a multiplicative weight on the GRPO gradient: the reward determines the update direction, and the KL determines its intensity at each token.
Figure 2 and Algorithm 1 illustrate the method. The core mechanism is described in § 5.1 : we construct a self-conditioned teacher by conditioning the model on a reference trajectory, compute token-level KL, and use it to weight the GRPO gradient. The reference is selected based on the number of correct rollouts n c n_{c} in the group: verified trajectories for partial-solve groups ( 2 ≤ n c < G 2\leq n_{c}<G ), random rollouts for solve-none groups ( n c = 0 n_{c}{=}0 ), and standard GRPO for n c = 1 n_{c}{=}1 or n c = G n_{c}{=}G . Section 5.2 analyzes the computational overhead.
### 5.1 Self-Conditioned GRPO
#### Teacher construction
For a rollout o i o_{i} with prefix states s i , t = ( x , o i , < t ) s_{i,t}=(x,o_{i,<t}) and a verified trajectory τ ∈ 𝒞 ( x ) \tau\in\mathcal{C}(x) , we instantiate the self-conditioned teacher π ~ θ ( ⋅ ∣ s i , t , τ ) \widetilde{\pi}_{\theta}(\cdot\mid s_{i,t},\tau) via Eq. ( 4 ), where τ \tau appears only in the teacher’s system prompt. Appendix E provide the complete prompt templates. Both teacher and student are scored on the same prefix o i , < t o_{i,<t} of o i o_{i} . At each response token, we compute the forward KL from teacher to student: D i , t = KL ( π ~ θ ( ⋅ ∣ s i , t , τ ) ∥ π θ ( ⋅ ∣ s i , t ) ) . D_{i,t}=\mathrm{KL}\!\left(\widetilde{\pi}_{\theta}(\cdot\mid s_{i,t},\tau)\;\middle\|\;\pi_{\theta}(\cdot\mid s_{i,t})\right).
#### KL weighting
We map each D i , t D_{i,t} to a bounded weight f ( D i , t ) ∈ [ 0 , 1 ) f(D_{i,t})\in[0,1) to modulate the GRPO gradient: tokens with small KL are downweighted, while tokens with large KL are preserved. We map each D i , t D_{i,t} to a bounded weight via f ( D i , t ) \displaystyle f(D_{i,t}) = D i , t D i , t + c , \displaystyle=\frac{D_{i,t}}{D_{i,t}+c}, c \displaystyle c = max ( P 75 ( 𝒟 act ) , c min ) , \displaystyle=\max\!\left(P_{75}(\mathcal{D}_{\mathrm{act}}),\,c_{\min}\right), where 𝒟 act \mathcal{D}_{\mathrm{act}} is the set of active token KL values in the current micro-batch, P 75 P_{75} denotes its 75th percentile, and c min > 0 c_{\min}>0 is a small floor. We use P 75 P_{75} instead of median because token-level KL is sparse (most tokens have near-zero KL). P 75 P_{75} ensures low-KL tokens are suppressed while high-KL tokens are preserved. We visualize and analyze the KL distribution and weighting function in Appendix C.3 and Appendix C.4 . A detailed case study is provided in Appendix D .
#### Group routing
The reference trajectory τ \tau and advantage A ^ i \hat{A}_{i} are selected based on the group’s solve pattern. Partial-solve groups have both correct and incorrect rollouts, allowing us to use verified trajectories as references. Solve-none groups have zero GRPO gradient, so we extract exploration signal via diversity.
Partial-solve groups ( 2 ≤ n c < G 2\leq n_{c}<G ): For each rollout o i o_{i} , we sample a verified trajectory τ ∈ 𝒞 ( x ) \tau\in\mathcal{C}(x) uniformly (with τ ≠ o i \tau\neq o_{i} when o i ∈ 𝒞 ( x ) o_{i}\in\mathcal{C}(x) ). We use the GRPO advantage A ^ i \hat{A}_{i} in Eq. ( 1 ).
Solve-none groups ( n c = 0 n_{c}=0 ): We sample one rollout o r o_{r} uniformly as reference. For each remaining rollout o i o_{i} ( i ≠ r i\neq r ), we compute KL against the reference-conditioned teacher. The average KL weight over the reference-length prefix becomes a diversity score: s i = 1 L i ∑ t = 1 L i f ( D i , t ) , L i = min ( | o i | , | o r | ) , s_{i}=\frac{1}{L_{i}}\sum_{t=1}^{L_{i}}f(D_{i,t}),\quad L_{i}=\min(|o_{i}|,|o_{r}|), where L i L_{i} truncates at the reference length. Since the reference itself is a failed trajectory, we apply stricter supervision to prevent reward hacking: tokens beyond the reference would have artificially high KL and inflate diversity scores for longer rollouts. Scores are normalized within the group to form a pseudo-advantage: A ^ i div = s i − mean ( { s k } k = 1 G ) std ( { s k } k = 1 G ) + ϵ , \hat{A}^{\mathrm{div}}_{i}=\frac{s_{i}-\mathrm{mean}(\{s_{k}\}_{k=1}^{G})}{\mathrm{std}(\{s_{k}\}_{k=1}^{G})+\epsilon}, (5) which replaces A ^ i \hat{A}_{i} with α A ^ i div \alpha\hat{A}^{\mathrm{div}}_{i} in the SC-GRPO objective. Since all rollouts failed which bring zero GRPO advantage, this diversity signal encourages exploring new modes rather than repeating common ones.
Fallback ( n c = 1 n_{c}=1 or n c = G n_{c}=G ): Standard GRPO (no KL weighting).
SC-GRPO optimizes the weighted GRPO objective: J SC - GRPO ( θ ) = 𝔼 [ 1 G ∑ i = 1 G 1 | o i | ∑ t = 1 | o i | f ( D i , t ) min ( ρ i , t A ^ i , clip ( ρ i , t , 1 − ϵ , 1 + ϵ ) A ^ i ) ] , \begin{split}&J_{\mathrm{SC\text{-}GRPO}}(\theta)=\mathbb{E}\Big[\tfrac{1}{G}\!\sum_{i=1}^{G}\tfrac{1}{|o_{i}|}\!\sum_{t=1}^{|o_{i}|}\\ &\quad f(D_{i,t})\,\min\!\big(\rho_{i,t}\hat{A}_{i},\,\mathrm{clip}(\rho_{i,t},1{-}\epsilon,1{+}\epsilon)\hat{A}_{i}\big)\Big],\end{split}\vskip-5.69054pt where ρ i , t = π θ ( o i , t ∣ s i , t ) / π θ old ( o i , t ∣ s i , t ) \rho_{i,t}=\pi_{\theta}(o_{i,t}\mid s_{i,t})/\pi_{\theta_{\mathrm{old}}}(o_{i,t}\mid s_{i,t}) is the importance ratio, A ^ i \hat{A}_{i} is the advantage (GRPO advantage from Eq. ( 1 ) for partial-solve groups, α A ^ i div \alpha\hat{A}^{\mathrm{div}}_{i} from Eq. ( 5 ) for solve-none groups). The KL weight f ( D i , t ) f(D_{i,t}) is applied symmetrically: both correct and incorrect rollouts are downweighted at low-KL tokens, regardless of their advantage sign.
### 5.2 Computational Overhead
SC-GRPO adds a single new operation compared to GRPO: a teacher forward pass for per-token KL weighting. As shown in Figure 3 , this makes the actor update 51% slower on LiveCodeBench and 37% slower on DAPO-Math, but end-to-end step time grows by only 13% and 2% respectively, since rollout generation dominates total latency. Combined with the gains in Section 6.3 , this confirms that SC-GRPO’s improvements come from better use of existing rollouts rather than additional compute.
## 6 Experiments
### 6.1 Setup
#### Tasks
We train and evaluate on RLVR tasks spanning mathematical reasoning, code generation, and multi-turn agentic interaction. ( i ) Math : we train on DAPO-Math-17k ( Yu et al., 2026 ) and evaluate on AIME 2024 & 2025. ( ii ) Code : we use LiveCodeBench v6 ( Jain et al., 2025 ) (LCB), holding out half of each problem’s unit tests for training and using the rest for evaluation. ( iii ) AppWorld ( Trivedi et al., 2024 ) , ( iv ) WebShop ( Yao et al., 2022 ) . For both AppWorld and WebShop, we train and evaluated on the official splits. All runs share the same base model, Qwen3-8B with thinking disabled ( Yang et al., 2025 ) . Full per-task settings are detailed in Appendix B.1 .
#### Metrics
For each query we sample eight trajectories and report two metrics: Avg@8 , the mean verifier reward across the eight samples; and Pass@8 , the fraction of queries solved by at least one of the eight samples.
### 6.2 Baselines
We compare SC-GRPO against two families of baselines, all sharing the same base model, training data, rollouts-per-prompt, and step budget as our method.
#### Reinforcement learning
GRPO ( Guo et al., 2025 ) is the canonical RLVR algorithm. DAPO ( Yu et al., 2026 ) is a stronger GRPO variant incorporating decoupled clipping, dynamic sampling, and token-level loss normalization. REINFORCE++ ( Hu et al., 2025 ) simplifies PPO by removing the critic and shaping rewards with per-token KL penalties.
#### OPSD with external demonstrations
Following Zhao et al. (2026) , we report OPSD baselines that supply the teacher with demonstrations τ \tau produced by external LLMs and train student with the same OPSD objective. We consider three demonstration sources of progressively higher quality: MiniMax-M2.7 1 1 1 https://huggingface.co/MiniMaxAI/MiniMax-M2.7 , DeepSeekv4-Pro DeepSeek-AI (2026) , and Oracle , which obtain as many correct solutions as possible, simulating access to all reference solutions. Details and prompt templates are provided in Appendix B.2 .
### 6.3 Main Results
SC-GRPO consistently outperforms all baselines. Across all five benchmarks (Table 1 ), SC-GRPO achieves the highest Avg@8 and Pass@8, outperforming DAPO by 5.86% and 8.92% respectively. The largest improvements appear on multi-turn agentic tasks, where credit assignment over long horizons is most challenging. We provide detailed training dynamics in Appendix C .
#### REINFORCE++ exhibits unstable performance across tasks.
Despite also operating at the token level, REINFORCE++ shows inconsistent results: competitive on code generation but significantly weaker on mathematical reasoning, and collapsing on multi-turn agentic tasks. We analyze the training dynamics in Appendix B.4 . This instability highlights that removing the critic sacrifices robustness across diverse task structures. In contrast, SC-GRPO remains stable across all five benchmarks.
External demonstrations do not reliably help. The three OPSD variants all stay close to the Qwen3-8B base model, substantially below the RL baselines. Unexpectedly, Oracle fails to outperform DeepSeekv4-Pro and even underperforms the base model on LCB, suggesting that demonstration quantity and quality are not the bottleneck. These results indicate that OPSD constructed from external demonstrations provides an unreliable training signal, as we demonstrate in § 4.2 . SC-GRPO addresses this by drawing the teacher trajectory from the model’s own verifier-approved rollouts, achieving substantially stronger performance.
## 7 Analysis
In this section, we conduct a comprehensive analysis to answer the following research questions: RQ1: Why must the KL signal be a weight rather than a loss? (§ 7.1 ) RQ2: Which design choices within SC-GRPO matter most? (§ 7.2 ) RQ3: Do the gains transfer out of domain? (§ 7.3 )
### 7.1 RQ1: Additive Loss
A natural alternative to SC-GRPO is to add the self-conditioned teacher’s KL divergence as an auxiliary distillation loss: ℒ GRPO + β ℒ distill \mathcal{L}_{\mathrm{GRPO}}+\beta\,\mathcal{L}_{\mathrm{distill}} . We test Forward KL, Reverse KL (both with fixed β = 0.1 \beta=0.1 and linear warm-up β : → 0.1 \beta:0.03\!\to\!0.1 ), and Bidirectional KL (combining both directions with complementary coefficients). Table 2 shows that all five variants underperform SC-GRPO.
Following Li et al. (2026) , we compare SC-GRPO against OPD under two configurations (Figure 4 ). The left panel (Qwen3-1.7B-Base student, Qwen3-4B-Base teacher) represents a failing OPD setting, the right panel (DeepSeek-R1-Distill-Qwen-1.5B ( Guo et al., 2025 ) student, JustRL-DeepSeek-1.5B ( He et al., 2025 ) teacher) represents a successful setting. Hyperparameters are detailed in Appendix B.3 . Figure 4 shows that SC-GRPO consistently outperforms OPD in both settings. When OPD fails (left), SC-GRPO achieves 23% improvement, when OPD succeeds (right), SC-GRPO still outperforms OPD on both AIME tasks, while requiring no external teacher.
### 7.2 RQ2: Ablation Study
We validate the three core design choices in SC-GRPO (Table 3 ).
#### A1 Group Routing
The KL weighting is designed to operate where meaningful contrast exists. Partial-solve groups ( 2 ≤ n c < G 2\leq n_{c}<G ) provide such contrast and benefit from KL weighting. Adding solve-none groups ( n c = 0 n_{c}=0 ) brings further improvement, as the diversity signal recovers useful updates from otherwise zero-gradient groups. Including single-correct groups ( n c = 1 n_{c}=1 ) violates the design premise: the single verified trajectory dominates the KL signal, causing entropy collapse. Including all-correct groups ( n c = G n_{c}=G ) similarly fails: without incorrect rollouts for contrast, self-conditioned teacher cannot separate critical decision points from routine tokens. Covering all groups combines both failure modes and drops below GRPO.
#### A2 Normalization threshold
The threshold c c separates the informative high-KL tail from the uninformative majority (visualization in Appendix C.3 ). When c = p 50 c=p_{50} (median), the heavily right-skewed KL distribution pushes c c below 10 − 4 10^{-4} , making f ( KL ) ≈ 1 f(\mathrm{KL})\approx 1 for nearly all tokens and reducing to uniform weighting. Using c = max ( p 75 , 10 − 4 ) c=\max(p_{75},10^{-4}) achieves the intended separation: p 75 p_{75} suppresses the low-KL majority while the floor prevents collapse when the entire batch has near-zero KL. A fixed constant achieves comparable performance but requires per-task calibration; p 75 p_{75} with floor is adaptive and hyperparameter-free.
#### A3 Diversity coefficient
The diversity signal rewards exploration rather than correctness, so it should serve as a weak exploration signal that does not override the reward gradient. The ablation confirms this: α = 0.1 \alpha{=}0.1 provides sufficient pressure to break repeated failure patterns, while α = 0.2 \alpha{=}0.2 lets the exploration signal compete with the RL objective and degrades performance.
### 7.3 RQ3: Out-of-Domain Performance
Both SC-GRPO and DAPO are trained exclusively on LiveCodeBench and evaluated on Codeforces without domain-specific fine-tuning. Figure 5 shows that SC-GRPO retains its advantage, achieving 5.2% relative gain on Avg@8 (nearly identical to the 6.1% in-domain gain on LCB v6) while maintaining comparable Pass@8. The consistent improvement across domains suggests that SC-GRPO’s gains stem from improved credit assignment rather than overfitting.
## 8 Conclusion
We propose SC-GRPO, a token-level credit assignment method for RLVR. By constructing a self-conditioned teacher from the model’s own verified rollouts and measuring token-level KL divergence, SC-GRPO assigns fine-grained credit to each token. This KL signal is used as a per-token weight on the GRPO gradient, eliminating sensitivity to divergence form choice and requiring no external resources beyond what RLVR already provides. Experiments show that SC-GRPO consistently outperforms GRPO and other baseline with minimal computational overhead, demonstrating the effectiveness and robustness of SC-GRPO.
### Limitations
Due to computational resource constraints, our experiments are limited to models up to 8B parameters and response lengths up to 12288 tokens. The effectiveness of SC-GRPO on larger models and longer responses remains to be explored. Similarly, resource limitations restrict our evaluation to models operating in standard reasoning mode. The applicability of SC-GRPO to extended thinking modes (e.g., chain-of-thought with explicit reasoning steps, multi-turn refinement) or structured output formats remains unexplored.
### Ethics Statement
This work focuses on improving token-level credit assignment in reinforcement learning with verifiable rewards. All LLMs, RL frameworks, and datasets used in our experiments are publicly available and used in accordance with their respective licenses. As a training algorithm, SC-GRPO does not introduce new ethical concerns
## Appendix A Derivation for Direct OPSD in RLVR
This section formalizes why directly adapting OPSD in the standard RLVR setting induces a token-level target mismatch. We fix a query x x , a student rollout prefix o i , < t o_{i,<t} , and analyze the induced distribution over the next token.
### A.1 Setup