Learning from Own Solutions: Self-Conditioned Credit Assignment for Reinforcement Learning with Verifiable Rewards — part 4/4

Cleaned-text code points [47265, 58859). Bibliography omitted here; source.md includes it.

All OPSD experiments use Qwen3-8B as the base model and train LoRA adapters with bfloat16 precision. Following ( Zhao et al., 2026 ) , we use LoRA rank r = 64 r=64 and scaling factor α = 128 \alpha=128 on all attention and MLP projection layers ( q_proj , k_proj , v_proj , o_proj , gate_proj , up_proj , and down_proj ). We optimize with learning rate 1 × 10 − 6 1\times 10^{-6} , gradient clipping at 1.0, per-device batch size 1, gradient accumulation 2, and 8 training processes, yielding an effective batch size of 16. For OPSD, we use a fixed teacher, β = 0 \beta=0 , λ = 1 \lambda=1 , temperature 1.0, top- p = 0.95 p=0.95 , and token-level JSD clipping threshold 0.05.

#### OPSD Prompt

The student prompt for each task follows that used in our main experiments. The teacher prompt injects a reference/answer block as privileged context. Below we show the teacher-specific prompt for each task type.

In all cases, the reference block is appended to the standard task prompt. The student never sees the reference block.

### B.3 OPD Details

To align with the settings in Li et al. (2026) , we use the same training and inference configurations as Li et al. (2026) in Figure 4 . See Table 6 for details. Due to computational resource constraints, we sample 3,600 examples for RL training, which corresponds to the 60 step performance reported in Li et al. (2026) .

### B.4 Analysis of Reinforce++

REINFORCE++ ( Hu et al., 2025 ) applies per-token credit assignment through discounted returns: G t = r t + γ ​ G t + 1 G_{t}=r_{t}+\gamma G_{t+1} . While competitive on short-horizon tasks (math, code), it collapses on long-horizon agentic tasks.

On AppWorld, validation performance peaked at 0.175@step40, then declined to 0.035@step101. On WebShop, performance dropped from 0.327 (base model) to 0.0@step80, with training reward also reaching 0.0 in the final steps.

The instability stems from gradient variance in long horizons. Early tokens accumulate returns from all subsequent tokens: G 1 = ∑ t = 1 T γ t − 1 ​ r t G_{1}=\sum_{t=1}^{T}\gamma^{t-1}r_{t} . With γ = 1.0 \gamma=1.0 and trajectories spanning hundreds of tokens, gradient variance grows exponentially. Combined with sparse episode-level rewards, advantage whitening fails to identify critical tokens, causing policy collapse.

In contrast, SC-GRPO uses trajectory-level GRPO advantages and applies token-level credit assignment only through KL-based gradient modulation, avoiding variance explosion in long-horizon settings.

## Appendix C Training Dynamics

### C.1 Training Dynamics on AIME 24 & 25

Figure 6 shows the validation performance throughout training on AIME 2024 and 2025. Both methods are trained on DAPO-Math-17k with identical hyperparameters and evaluated every 20 steps.

#### AIME 2024 (Top Row)

On Avg@8 (left), SC-GRPO establishes an early advantage and maintains it throughout training. Both methods exhibit some variance in the later stages of training, but SC-GRPO consistently stays above DAPO. On Pass@8 (right), the gap is more pronounced: SC-GRPO achieves 0.80 at step 280, while DAPO peaks at 0.72 around step 260 before declining slightly.

#### AIME 2025 (Bottom Row)

AIME 2025 is harder than AIME 2024, with both methods achieving lower absolute scores. On Avg@8 (left), SC-GRPO shows steady improvement throughout training, reaching 0.40 by step 500. The advantage is more consistent here than on AIME 2024, with less variance in the later stages. On Pass@8 (right), SC-GRPO maintains a 5-10% advantage throughout most of training, reaching 0.63 compared to DAPO’s 0.57.

#### Stability and convergence

Both methods exhibit training variance typical of RL with verifiable rewards, where validation performance fluctuates due to the discrete nature of the reward signal. However, SC-GRPO demonstrates more stable improvement on AIME 2025, the harder test set, suggesting that token-level credit assignment becomes increasingly valuable as problem difficulty increases. The consistent gap across both metrics and both test sets confirms that the gains are not due to overfitting to a particular evaluation protocol.

### C.2 Policy Entropy

Figure 7 shows the policy entropy throughout training. SC-GRPO maintains consistently higher entropy than DAPO across all training steps, with the gap widening after step 200. This suggests that SC-GRPO preserves exploration capacity: by selectively suppressing gradients on tokens where the student already matches the teacher and encouraging exploration when facing solve-none groups, SC-GRPO avoids premature convergence on the full trajectory. In contrast, DAPO’s uniform credit assignment drives the policy toward deterministic outputs more aggressively, reducing entropy and potentially limiting the model’s ability to explore alternative reasoning paths.

The entropy gap correlates with the performance advantage observed in Figure 6 : higher entropy enables the model to maintain diverse solution strategies, which is particularly valuable on harder problems (AIME 2025) where multiple reasoning approaches may be necessary to reach the correct answer.

### C.3 KL Weighting Function

Figure 8 illustrates why we use the 75th percentile of token-level KL divergence as the threshold c c in our weighting function. The function f ⁡ ( KL ) = KL / ( KL + c ) f(\text{KL})=\text{KL}/(\text{KL}+c) maps each token’s KL divergence to a weight between 0 and 1, which modulates the RL gradient for that token.

When c = p 25 c=p_{25} (orange curve), the threshold is near zero, so almost all tokens receive high weights ( f ⁡ ( KL ) ≈ 1 f(\text{KL})\approx 1 ), effectively reverting to uniform credit assignment. When c = p 90 c=p_{90} (red curve), the threshold is too high, causing many tokens with moderate KL divergence (0.05–0.15) to receive low weights, over-suppressing the gradient signal.

Our choice of c = p 75 c=p_{75} (blue curve) strikes a balance: tokens with KL below p 75 p_{75} (the bottom 75% of the distribution) receive weights below 0.5, indicating the student has already learned these tokens from the teacher. Only the top 25% of tokens, those with the highest KL divergence, receive weights above 0.5, preserving the RL gradient where the student and teacher genuinely differ. This adaptive threshold ensures that credit assignment focuses on critical difference tokens while avoiding both under-discrimination and over-suppression.

### C.4 KL Distribution Evolution

Figure 9 shows how the KL distribution evolves throughout training. The distribution exhibits a heavy-tailed structure: the bottom 75% of tokens (p25 to p75, dark blue region) have very low KL divergence, often near zero, indicating that the student policy has already learned to match the teacher on these tokens. Only the top 25% of tokens show significant divergence.

This distribution remains remarkably stable across training steps, with p75 fluctuating around 0.03–0.10 and p95 around 0.2–0.5. The stability validates our choice of using p75 as an adaptive threshold: it consistently identifies the boundary between learned tokens and critical difference tokens, regardless of training stage. The occasional spikes in the lower percentiles (visible as vertical streaks in the p25-p75 region) correspond to batches with particularly challenging problems where even common tokens require learning, but these are rare and do not affect the overall threshold selection.

## Appendix D Case Study: Token-Level KL Heatmap

Figure 10 shows the per-token KL distribution a real LiveCodeBench problem (counting the minimum number of edges to remove so that an undirected graph becomes a forest) . The student rollout o i o_{i} implements the standard Disjoint Set Union (DSU / Union-Find) approach, while the sampled verifier-approved trajectory τ \tau used to condition the teacher implements a stack-based Depth-First Search (DFS) over an adjacency list. Both solve the problem correctly, but follow distinct algorithmic paths.

Panels (a) show the sampled verifier-approved trajectory τ \tau that is injected into the teacher’s system prompt. Panel (b) shows the student rollout o i o_{i} , with each token shaded by its KL weight f t = D t / ( D t + c ) f_{t}=D_{t}/(D_{t}+c) . Near-white tokens correspond to positions where the student’s distribution is essentially unchanged by conditioning on τ \tau : the model would produce these tokens whether or not a verified solution was available. Deeply shaded tokens mark positions where τ \tau substantially redirects the next-token distribution, these are the tokens on which the GRPO update is preserved at full weight, while the near-white tokens are downweighted.

#### What tokens are selected

Most low-level scaffolding code (indentation, parentheses, simple variable assignments, generic loops) receives near-zero weight, since both DSU and DFS share these surface forms. The high-KL tokens concentrate at the points where the two algorithms genuinely diverge: the import of collections and the choice of stdin reader; the allocation of the DSU parent = list(range(N+1)) array (absent under DFS, which builds an adjacency list instead); the bodies of def find and def union , the merge predicate root_u != root_v , and the construction of the components set; and finally the closed-form answer answer = M - (N - K) . These are precisely the tokens whose choice distinguishes a DSU implementation from a DFS one, and on which a verified DFS trajectory shifts the student’s belief most strongly.

The pattern is consistent with the design intent of SC-GRPO: the self-teacher does not act as an additional supervision signal, it acts as a credit-assignment filter on the existing GRPO gradient. Sequence-level reward is preserved as the source of update direction, while gradient mass is concentrated on the small subset of tokens that actually carry the algorithmic decision.

## Appendix E Prompt Templates for SC-GRPO

### E.1 Prompt Construction

For a given problem instance, we construct student and teacher prompts as follows: The student prompt consists of a base system prompt followed by the problem specification. The teacher prompt extends this by concatenating three components: the base system prompt, the guide instruction, and the reference trajectory.

### E.2 LiveCodeBench Templates Example

We use LiveCodeBench as an example to illustrate our prompt templates. The same structure applies to other tasks (math and agentic tasks) with domain-specific guide instructions.

For math tasks (DAPO-Math-17K) and agentic tasks (AppWorld & Webshop), we use task-specific system prompts and guide instructions.

## Instructions for reporting errors

We are continuing to improve HTML versions of papers, and your feedback helps enhance accessibility and mobile support. To report errors in the HTML that will help us improve conversion and rendering, choose any of the methods listed below:

Click the "Report Issue" ( ) button, located in the page header.

Tip: You can select the relevant text first, to include it in your report.

Our team has already identified the following issues . We appreciate your time reviewing and reporting rendering errors we may not have found yet. Your efforts will help us improve the HTML versions for all readers, because disability should not be a barrier to accessing research. Thank you for your continued support in championing open access for all.

Have a free development cycle? Help support accessibility at arXiv! Our collaborators at LaTeXML maintain a list of packages that need conversion , and welcome developer contributions .