Learning from Own Solutions: Self-Conditioned Credit Assignment for Reinforcement Learning with Verifiable Rewards — part 1/4

Cleaned-text code points [0, 15761). Bibliography omitted here; source.md includes it.

# Learning from Own Solutions: Self-Conditioned Credit Assignment for Reinforcement Learning with Verifiable Rewards

###### Abstract

Reinforcement learning with verifiable rewards (RLVR) has driven substantial progress in training LLMs for reasoning tasks, but representative methods such as GRPO assign uniform credit across all tokens, wasting gradient on routine tokens while under-crediting pivotal reasoning steps. Existing token-level credit assignment methods require resources beyond the model’s own rollouts. GRPO variants rely on process reward models or ground-truth answers. Knowledge distillation assigns credit through per-token divergence but requires external teachers (On-Policy Distillation) or privileged information (On-Policy Self Distillation). However, these dependencies limit applicability in the pure RLVR setting. We observe that conditioning the model on its own verified trajectories induces a measurable per-token KL divergence between the original and conditioned distributions, and prove that distilling from a self-teacher constructed by verified trajectories leads to infeasible weighted-average solutions when multiple verified trajectories exist. We propose SC-GRPO ( S elf- C onditioned GRPO), which uses KL divergence mentioned before as a multiplicative weight on GRPO gradients. Across five benchmarks spanning math, code, and agentic tasks, SC-GRPO consistently outperforms 8.1% over GRPO and 5.9% over DAPO with stronger OOD performance. Moreover, SC-GRPO achieves higher performance than OPD.

## 1 Introduction

Large language models (LLMs) have shown strong capabilities in various tasks, but competition-level problems remain challenging. Reinforcement learning with verifiable rewards (RLVR) ( Guo et al., 2025 ; Yu et al., 2026 ; He et al., 2026a ) has driven substantial progress in tackling such problems. Representative methods in RLVR including GRPO Guo et al. (2025) ; Liu et al. () ; Zheng et al. (2025) ; Wang et al. (2025c) assign a single scalar reward per rollout, so every token shares the same advantage. This uniform credit cannot identify which tokens caused success or failure, diluting gradient across routine tokens while under-crediting pivotal reasoning steps ( He et al., 2026b ; Xie et al., 2025 ) .

To enable token-level credit assignment, existing approaches require resources beyond the model’s own rollouts (Figure 1 , top). Recent GRPO variants have explored different directions: some train process reward models on step-level annotations ( Wang et al., 2024b ; Cui et al., 2025 ) , while others rely on ground-truth answers ( Wang et al., 2025b ) . Alternatively, knowledge distillation directly assigns credit at the token level through per-position divergence between student and teacher distributions Hinton et al. (2015) . On-Policy Distillation (OPD) ( Agarwal et al., 2024 ; Ko et al., 2024 ) uses a stronger external teacher to provide token-level supervision, and On-Policy Self-Distillation (OPSD) ( Hübotter et al., 2026 ; Zhao et al., 2026 ) similarly requires privileged information. These dependencies limit applicability in the pure RLVR setting, where only a binary verifier is available.

However, we can identify which tokens carry credit by comparing predictions with and without access to the model’s own verified solution. Recent work has shown that conditioning models on their own verified trajectories improves generation quality ( Yao et al., 2026 ) . Building on this observation, we propose SC-GRPO ( S elf- C onditioned GRPO), which uses such conditioning to induce a measurable per-token distributional shift as a credit assignment signal. As shown in Figure 1 bottom, for problems where the model produces both correct and incorrect rollouts, SC-GRPO injects one verified correct solution as an in-context demonstration and re-scores all remaining rollouts under this prompt, constructing a self-conditioned teacher. A complete version of this example is provided in Figure 10 .

We quantify this shift via per-token Kullback–Leibler (KL) divergence between the original and demonstration-conditioned next-token distributions. Inspired by OPSD ( Zhao et al., 2026 ) , a natural approach would be to add this KL as an auxiliary distillation loss. However, when multiple distinct verified trajectories exist, the loss-minimizing student converges to a weighted average over these teachers, a distribution that does correspond to any feasible trajectory (we formalize this in § 4.2 ). Empirically, such additive formulations consistently fail to improve over GRPO (§ 7.1 ).

Instead, SC-GRPO uses the KL signal purely as a multiplicative weight on the GRPO gradient: the reward determines the update direction, and the KL determines only its intensity at each token. The same self-conditioned teacher handles both partial-solve groups, where a verified correct trajectory serves as the reference and the KL weight sharpens credit assignment, and solve-none groups, where a random rollout serves as the reference and the KL signal provides a slight exploration bonus that encourages diversity. On five benchmarks spanning math, code, and multi-turn agentic tasks, SC-GRPO improves Average@8 by 8.1% over GRPO and 5.9% over DAPO, while achieving more consistent and higher performance than OPD with external teachers.

Our contributions are as follows: • In RLVR, verified trajectories are the only supervision beyond the binary verifier. We prove that distilling from a teacher constructed using these trajectories leads to infeasible solutions, and demonstrate empirically that such formulations fail to improve over GRPO.

• We propose SC-GRPO, which constructs a teacher by conditioning the model on its own verified trajectories and uses the resulting token-level KL divergence as a multiplicative weight on GRPO gradients.

• Across five benchmarks spanning math, code, and agentic tasks, SC-GRPO consistently outperforms RL baselines and achieves higher and more stable performance than OPD.

## 2 Related Work

### 2.1 On-Policy Distillation

On-policy distillation trains on student generated rollouts with a divergence to the teacher distribution ( Agarwal et al., 2024 ; Gu et al., 2024 ; Ko et al., 2024 ; Shing et al., ) , eliminating the train-test mismatch of offline distillation ( Hinton et al., 2015 ) . The divergence objective has been refined through skew-KL ( Ko et al., 2024 ) , α \alpha - β \beta -divergence ( Wang et al., 2025a ) , and adaptive interpolation ( Shing et al., ) , highlighting the sensitivity of letting divergence set the gradient direction. Moreover, all variants require a stronger external teacher.

Self-distillation removes the external teacher dependency by constructing a teacher from the model itself, typically conditioned on privileged information such as textual feedback ( Hübotter et al., 2026 ) , ground truth ( Zhao et al., 2026 ; Ding, 2026 ; Yang et al., 2026 ) , or expert demonstrations ( Shenfeld et al., 2026 ) . SC-GRPO uses KL as a per-token weight, eliminating sensitivity to divergence form choice, and conditions the teacher solely on the model’s own verified rollouts.

### 2.2 Token-Level Credit Assignment in RL

Process reward models (PRMs) ( Lightman et al., 2023 ; Wang et al., 2024b ; Wang et al., 2024a ) provide step-level supervision but require training a separate reward model with large-scale annotation, and remain vulnerable to reward hacking ( Juneja et al., 2025 ; Gao et al., 2024 ) . Annotation-free alternatives estimate token values through Monte Carlo sampling ( Guan et al., ) or segment-level sampling ( Guo et al., 2026 ) , requiring additional rollout compute. GRPO-based methods also need external PRM models Cui et al. (2025) or ground truth Wang et al. (2025b) to gain token-level credit during training. In contrast, SC-GRPO only depends on the model’s own rollouts to obtain token-level reward without external resources.

## 3 Preliminaries

#### Notation

Let π θ \pi_{\theta} denote the policy parameterized by θ \theta . Given a query x x sampled from the training dataset 𝒟 \mathcal{D} , the policy generates a response o = ( o 1 , … , o | o | ) ∼ π θ ( ⋅ ∣ x ) o=(o_{1},\ldots,o_{|o|})\sim\pi_{\theta}(\cdot\mid x) , where o t o_{t} is the token at position t t and o < t o_{<t} denotes the prefix before position t t . Each response is evaluated by a verifier r r , yielding a scalar reward r ⁡ ( x , o ) ∈ [ 0 , 1 ] r(x,o)\in[0,1] .

### 3.1 GRPO

For each query x x , GRPO samples a group of G G responses { o i } i = 1 G \{o_{i}\}_{i=1}^{G} from the old policy π θ old \pi_{\theta_{\mathrm{old}}} . Each response is evaluated by the verifier, yielding r i = r ⁡ ( x , o i ) r_{i}=r(x,o_{i}) . Instead of learning a value function, GRPO normalizes rewards across the G G responses in the same group to obtain a sequence-level advantage. Equivalently, every token position t t in response o i o_{i} shares the same advantage: A ^ i , t = A ^ i = r ⁡ ( x , o i ) − mean ⁡ ( { r ⁡ ( x , o k ) } k = 1 G ) std ⁡ ( { r ⁡ ( x , o k ) } k = 1 G ) . \hat{A}_{i,t}=\hat{A}_{i}=\frac{r(x,o_{i})-\mathrm{mean}(\{r(x,o_{k})\}_{k=1}^{G})}{\mathrm{std}(\{r(x,o_{k})\}_{k=1}^{G})}. (1) The policy is then updated with following objective over response tokens:

𝒥 GRPO ( θ ) = 𝔼 x ∼ 𝒟 , { o i } i = 1 G ∼ π θ old ( ⋅ ∣ x ) [ 1 G ∑ i = 1 G 1 | o i | \displaystyle\mathcal{J}_{\mathrm{GRPO}}(\theta)=\mathbb{E}_{x\sim\mathcal{D},\,\{o_{i}\}_{i=1}^{G}\sim\pi_{\theta_{\mathrm{old}}}(\cdot\mid x)}\Bigg[\frac{1}{G}\sum_{i=1}^{G}\frac{1}{|o_{i}|} ∑ t = 1 | o i | min ( ρ i , t A ^ i , t , clip ( ρ i , t , 1 − ϵ , 1 + ϵ ) A ^ i , t ) ] , \displaystyle\sum_{t=1}^{|o_{i}|}\min\!\bigl(\rho_{i,t}\hat{A}_{i,t},\,\mathrm{clip}(\rho_{i,t},1-\epsilon,1+\epsilon)\hat{A}_{i,t}\bigr)\Bigg], with ρ i , t = π θ ​ ( o i , t ∣ x , o i , < t ) π θ old ​ ( o i , t ∣ x , o i , < t ) . \displaystyle\mathrm{with}\quad\rho_{i,t}=\frac{\pi_{\theta}(o_{i,t}\mid x,o_{i,<t})}{\pi_{\theta_{\mathrm{old}}}(o_{i,t}\mid x,o_{i,<t})}.

### 3.2 On-Policy Distillation

OPD Agarwal et al. (2024) provides dense token-level signal along the student’s own trajectories.

Let π θ \pi_{\theta} and π T \pi_{T} denote the student and frozen teacher policies, respectively. Given a prompt x ∼ 𝒟 x\sim\mathcal{D} , the student samples a response o ∼ π θ ( ⋅ ∣ x ) o\sim\pi_{\theta}(\cdot\mid x) , which induces prefix states s t = ( x , o < t ) s_{t}=(x,o_{<t}) . At each prefix, the student and teacher next-token distributions are π θ ( ⋅ ∣ s t ) \pi_{\theta}(\cdot\mid s_{t}) and π T ( ⋅ ∣ s t ) \pi_{T}(\cdot\mid s_{t}) , and the token-level OPD objective is ℒ OPD ​ ( θ ) \displaystyle\mathcal{L}_{\mathrm{OPD}}(\theta) = 𝔼 x ∼ 𝒟 , o ∼ π θ ( ⋅ ∣ x ) [ 1 | o | ∑ t = 1 | o | \displaystyle=\mathbb{E}_{\,x\sim\mathcal{D},\;o\sim\pi_{\theta}(\cdot\mid x)}\Big[\frac{1}{|o|}\sum_{t=1}^{|o|} (2) D ( π T ( ⋅ ∣ s t ) ∥ π θ ( ⋅ ∣ s t ) ) ] , \displaystyle D\!\bigl(\pi_{T}(\cdot\mid s_{t})\,\|\,\pi_{\theta}(\cdot\mid s_{t})\bigr)\Big], where D D is a divergence such as the Kullback–Leibler (KL) divergence or the Jensen–Shannon divergence (JSD).

OPSD Ding (2026) ; Hübotter et al. (2026) ; Zhao et al. (2026) removes the requirement of external teacher by conditioning the same model on privileged information c c (e.g., ground-truth or expert demonstration). The student distribution remains π θ ( ⋅ ∣ s t ) \pi_{\theta}(\cdot\mid s_{t}) , while the self-teacher is instantiated as: π ~ θ ( ⋅ ∣ s t , c ) := sg [ π θ ( ⋅ ∣ x , c , o < t ) ] , \widetilde{\pi}_{\theta}(\cdot\mid s_{t},c)\;:=\;\operatorname{sg}\!\left[\pi_{\theta}(\cdot\mid x,c,o_{<t})\right], (3) where sg ⁡ [ ⋅ ] \operatorname{sg}[\cdot] denotes the stop-gradient operator, indicating that the teacher distribution is treated as a fixed target for the current update. The training objective keeps the same form as Eq. ( 2 ), with π T ( ⋅ ∣ s t ) \pi_{T}(\cdot\mid s_{t}) replaced by π ~ θ ( ⋅ ∣ s t , c ) \widetilde{\pi}_{\theta}(\cdot\mid s_{t},c) .

## 4 Why Self-Distillation Fails in RLVR

### 4.1 Available Supervision in RLVR

For a query x x , GRPO samples a group of responses { o i } i = 1 G ∼ π θ old ( ⋅ ∣ x ) \{o_{i}\}_{i=1}^{G}\sim\pi_{\theta_{\mathrm{old}}}(\cdot\mid x) , and a verifier assigns a sequence-level reward r i = r ⁡ ( x , o i ) ∈ { 0 , 1 } r_{i}=r(x,o_{i})\in\{0,1\} to each response. Let 𝒞 ( x ) = { o i : r ( x , o i ) = 1 , i ∈ { 1 , … , G } } \mathcal{C}(x)=\{\,o_{i}:r(x,o_{i})=1,\ i\in\{1,\dots,G\}\,\} denote the set of verifier-approved trajectories.

In conventional self-distillation mentioned before, the teacher is conditioned on some form of privileged information. In the RLVR setting, however, no such privileged information is available. Beyond the query x x and the scalar reward r r , the only additional information we can access is a verifier-approved trajectory drawn from the model’s own rollouts, i.e., a sample from 𝒞 ⁡ ( x ) \mathcal{C}(x) . A verified trajectory is not privileged information. It certifies only end-task correctness, not reasoning quality. The trajectory may contain exploratory steps, redundant derivations, or recoverable mistakes, and when multiple trajectories in 𝒞 ⁡ ( x ) \mathcal{C}(x) succeed, they may follow contradictory reasoning paths.

### 4.2 Why OPSD Fails in RLVR

Despite this limitation, verified trajectories are the only additional structure available in RLVR. A natural but naive adaptation is therefore to treat them as a substitute for privileged context. Concretely, for a student rollout o i o_{i} with prefix states s i , t = ( x , o i , < t ) s_{i,t}=(x,o_{i,<t}) and a verifier-approved trajectory τ j ∈ 𝒞 ⁡ ( x ) \tau_{j}\in\mathcal{C}(x) , we instantiate the generic OPSD formulation in Section 3 by taking τ j \tau_{j} as the privileged context: π ~ θ ( ⋅ ∣ s i , t , τ j ) := sg [ π θ ( ⋅ ∣ x , τ j , o i , < t ) ] . \widetilde{\pi}_{\theta}(\cdot\mid s_{i,t},\,\tau_{j})\;:=\;\operatorname{sg}\!\left[\pi_{\theta}(\cdot\mid x,\,\tau_{j},\,o_{i,<t})\right]. (4) The training objective keeps the same form as Eq. ( 2 ), replacing the external teacher π T \pi_{T} with the self-conditioned teacher π ~ θ ( ⋅ ∣ s i , t , τ j ) \widetilde{\pi}_{\theta}(\cdot\mid s_{i,t},\,\tau_{j}) .

This adaptation, however, does not work. A pointwise analysis (detailed in Appendix A ) shows that under both forward and reverse KL, the loss-minimizing student distribution at any prefix s i , t s_{i,t} is a aggregate of the self-conditioned teachers { p j } \{p_{j}\} across 𝒞 ⁡ ( x ) \mathcal{C}(x) : q ⋆ ∝ { ∑ j μ x ​ ( j ) ​ p j (Forward KL), ∏ j p j μ x ​ ( j ) (Reverse KL), \!\!q^{\star}\propto\begin{cases}\sum_{j}\mu_{x}(j)\,p_{j}&\text{(Forward KL),}\\[2.0pt] \prod_{j}p_{j}^{\,\mu_{x}(j)}&\text{(Reverse KL),}\end{cases} where p j := π ~ θ ( ⋅ ∣ s i , t , τ j ) p_{j}:=\widetilde{\pi}_{\theta}(\cdot\mid s_{i,t},\tau_{j}) and μ x \mu_{x} is the selection rule over 𝒞 ⁡ ( x ) \mathcal{C}(x) . This average distribution is problematic: (i) it need not correspond to any single feasible trajectory; (ii) it assigns equal weight to all prefixes, even those where verified trajectories disagree about the next token; and (iii) it inherits whatever each τ j \tau_{j} happens to take to reach the correct answer. These issues are fundamental to any distillation-based approach that treats the teacher as a target distribution.

## 5 SC-GRPO

#### Overview

Section 4.2 shows that directly using verified trajectories as OPSD distillation targets fails in RLVR. Therefore, we propose SC-GRPO ( S elf- C onditioned GRPO), which takes a different approach. We construct a self-conditioned teacher by conditioning the model on a verified trajectory, then measure the per-token distributional shift between the teacher and the original student.