Learning from Own Solutions: Self-Conditioned Credit Assignment for Reinforcement Learning with Verifiable Rewards — part 3/4

Cleaned-text code points [31676, 47265). Bibliography omitted here; source.md includes it.

Following Section 3 , for a query x x let { o i } i = 1 G ∼ π θ old ( ⋅ ∣ x ) \{o_{i}\}_{i=1}^{G}\sim\pi_{\theta_{\mathrm{old}}}(\cdot\mid x) be the sampled response group, and let r ⁡ ( x , o i ) ∈ { 0 , 1 } r(x,o_{i})\in\{0,1\} be the verifier reward. The set of verifier-approved responses is 𝒞 ( x ) := { o i : r ( x , o i ) = 1 , i = 1 , … , G } . \mathcal{C}(x)\;:=\;\{\,o_{i}:r(x,o_{i})=1,\ i=1,\ldots,G\,\}. (A.1) Fix a student response o i o_{i} and token position t t , and let the prefix state be s i , t := ( x , o i , < t ) s_{i,t}:=(x,o_{i,<t}) . The student next-token distribution is π θ ( ⋅ ∣ s i , t ) = π θ ( ⋅ ∣ x , o i , < t ) . \pi_{\theta}(\cdot\mid s_{i,t})\;=\;\pi_{\theta}(\cdot\mid x,o_{i,<t}). (A.2) For each verifier-approved response τ j ∈ 𝒞 ⁡ ( x ) \tau_{j}\in\mathcal{C}(x) , direct OPSD constructs a self-conditioned teacher by additionally conditioning the same model on τ j \tau_{j} : π ~ θ ( ⋅ ∣ s i , t , τ j ) := sg [ π θ ( ⋅ ∣ x , τ j , o i , < t ) ] . \widetilde{\pi}_{\theta}(\cdot\mid s_{i,t},\,\tau_{j})\;:=\;\operatorname{sg}\!\left[\pi_{\theta}(\cdot\mid x,\,\tau_{j},\,o_{i,<t})\right]. (A.3) Let μ x \mu_{x} denote a selection distribution over 𝒞 ⁡ ( x ) \mathcal{C}(x) that specifies which verifier-approved response is used as the privileged context (e.g., uniform over 𝒞 ⁡ ( x ) \mathcal{C}(x) ).

The stop-gradient operator sg ⁡ [ ⋅ ] \operatorname{sg}[\cdot] indicates that the self-conditioned teacher is treated as a fixed target for the current update, and gradients do not backpropagate through teacher logits. We further assume all distributions are defined over a common finite vocabulary 𝒱 \mathcal{V} with shared support, which is satisfied by standard softmax LLM outputs.

### A.2 Forward-KL Direct OPSD

We use forward KL as a representative distillation objective and analyze direct OPSD pointwise at a single prefix.

#### Notation.

Fix a query x x , a student rollout o i o_{i} , and a token position t t , with prefix state s i , t = ( x , o i , < t ) s_{i,t}=(x,o_{i,<t}) . Throughout this subsection we suppress the dependence on ( i , t ) (i,t) and write q \displaystyle q := π θ ( ⋅ ∣ s i , t ) , \displaystyle:=\;\pi_{\theta}(\cdot\mid s_{i,t}), (A.4) p j \displaystyle p_{j} := π ~ θ ( ⋅ ∣ s i , t , τ j ) , τ j ∈ 𝒞 ( x ) . \displaystyle:=\;\widetilde{\pi}_{\theta}(\cdot\mid s_{i,t},\,\tau_{j}),\quad\tau_{j}\in\mathcal{C}(x). Here q q is the student next-token distribution and each p j p_{j} is a self-conditioned teacher distribution (treated as a fixed target via stop-gradient). All quantities are distributions over the vocabulary 𝒱 \mathcal{V} , for a distribution p p and a token y ∈ 𝒱 y\in\mathcal{V} , p ⁡ ( y ) ∈ [ 0 , 1 ] p(y)\in[0,1] denotes the probability assigned to y y .

The direct OPSD loss at this prefix is ℓ OPSD ( q ) := 𝔼 j ∼ μ x [ D KL ( p j ∥ q ) ] , \ell_{\mathrm{OPSD}}(q)\;:=\;\mathbb{E}_{j\sim\mu_{x}}\!\left[D_{\mathrm{KL}}\!\bigl(p_{j}\,\|\,q\bigr)\right], (A.5) and the μ x \mu_{x} -averaged teacher distribution is the per-token weighted average p ¯ ​ ( y ) := ∑ j μ x ​ ( j ) ​ p j ​ ( y ) , y ∈ 𝒱 . \bar{p}(y)\;:=\;\sum_{j}\mu_{x}(j)\,p_{j}(y),\qquad y\in\mathcal{V}. (A.6)

Inserting p ¯ ​ ( y ) \bar{p}(y) into the log ratio in Eq. ( A.5 ) and using 𝔼 j ​ [ p j ​ ( y ) ] = p ¯ ​ ( y ) \mathbb{E}_{j}[p_{j}(y)]=\bar{p}(y) yields ℓ OPSD ​ ( q ) \displaystyle\ell_{\mathrm{OPSD}}(q) = 𝔼 j ∼ μ x ​ ∑ y ∈ 𝒱 p j ​ ( y ) ​ log ⁡ p j ​ ( y ) q ⁡ ( y ) \displaystyle=\mathbb{E}_{j\sim\mu_{x}}\sum_{y\in\mathcal{V}}p_{j}(y)\,\log\tfrac{p_{j}(y)}{q(y)} (A.7) = 𝔼 j ∼ μ x ​ ∑ y ∈ 𝒱 p j ​ ( y ) ​ log ⁡ p j ​ ( y ) p ¯ ​ ( y ) \displaystyle=\mathbb{E}_{j\sim\mu_{x}}\sum_{y\in\mathcal{V}}p_{j}(y)\,\log\tfrac{p_{j}(y)}{\bar{p}(y)} + 𝔼 j ∼ μ x ∑ y ∈ 𝒱 p j ( y ) log p ¯ ​ ( y ) q ⁡ ( y ) \displaystyle+\mathbb{E}_{j\sim\mu_{x}}\sum_{y\in\mathcal{V}}p_{j}(y)\,\log\tfrac{\bar{p}(y)}{q(y)} = 𝔼 j ∼ μ x D KL ( p j ∥ p ¯ ) \displaystyle=\mathbb{E}_{j\sim\mu_{x}}\,D_{\mathrm{KL}}\!\bigl(p_{j}\,\|\,\bar{p}\bigr) + ∑ y ∈ 𝒱 p ¯ ( y ) log p ¯ ​ ( y ) q ⁡ ( y ) \displaystyle+\sum_{y\in\mathcal{V}}\bar{p}(y)\,\log\tfrac{\bar{p}(y)}{q(y)} = 𝔼 j ∼ μ x D KL ( p j ∥ p ¯ ) ⏟ 𝒟 disagree ≥ 0 + D KL ( p ¯ ∥ q ) . \displaystyle=\underbrace{\mathbb{E}_{j\sim\mu_{x}}\,D_{\mathrm{KL}}\!\bigl(p_{j}\,\|\,\bar{p}\bigr)}_{\mathcal{D}_{\mathrm{disagree}}\,\geq\,0}\;+\;D_{\mathrm{KL}}\!\bigl(\bar{p}\,\|\,q\bigr). where the expectation is over j ∼ μ x j\sim\mu_{x} and the sums are over y ∈ 𝒱 y\in\mathcal{V} . The first term 𝒟 disagree ≥ 0 \mathcal{D}_{\mathrm{disagree}}\geq 0 quantifies how much the self-conditioned teachers { p j } \{p_{j}\} disagree with each other at this prefix: it is zero exactly when all p j p_{j} coincide, and grows as they spread apart. Equivalently, when μ x \mu_{x} is uniform, 𝒟 disagree \mathcal{D}_{\mathrm{disagree}} is the generalized Jensen–Shannon divergence of { p j } \{p_{j}\} .

The disagreement term 𝒟 disagree \mathcal{D}_{\mathrm{disagree}} measures how much the self-conditioned teachers disagree at this prefix, crucially, it is constant in q q . The only q q -dependent term is D KL ( p ¯ ∥ q ) ≥ 0 D_{\mathrm{KL}}(\bar{p}\,\|\,q)\geq 0 , with equality if and only if q = p ¯ q=\bar{p} . Hence the unconstrained pointwise minimizer of Eq. ( A.5 ) is q ⋆ = p ¯ . q^{\star}\;=\;\bar{p}. (A.8)

Direct forward-KL OPSD does not target any particular verified trajectory’s local continuation, it targets the average p ¯ \bar{p} of the self-conditioned teacher distributions. The disagreement term 𝒟 disagree \mathcal{D}_{\mathrm{disagree}} shows that different verified trajectories can induce genuinely different local continuations, but this disagreement does not influence the optimizer because it is constant in q q . In a parametric policy class, the update can be viewed as projecting the student toward this average, an average that need not correspond to any single feasible trajectory.

### A.3 Reverse-KL Variant

The previous analysis is specific to forward KL. We now repeat the pointwise analysis for the reverse-KL variant of direct OPSD, using the same notation ( q , p j , μ x q,p_{j},\mu_{x} ) as in Appendix A.2 .

The reverse-KL direct OPSD loss at the prefix s i , t s_{i,t} is ℓ OPSD ( q ) := 𝔼 j ∼ μ x [ D KL ( q ∥ p j ) ] . \ell_{\mathrm{OPSD}}(q)\;:=\;\mathbb{E}_{j\sim\mu_{x}}\!\left[D_{\mathrm{KL}}\!\bigl(q\,\|\,p_{j}\bigr)\right]. (A.9)

Unfolding the KL by definition and using the linearity of expectation in j j : ℓ OPSD ​ ( q ) \displaystyle\ell_{\mathrm{OPSD}}(q) = 𝔼 j ​ ∑ y ∈ 𝒱 q ⁡ ( y ) ​ log ⁡ q ⁡ ( y ) p j ​ ( y ) \displaystyle=\mathbb{E}_{j}\sum_{y\in\mathcal{V}}q(y)\,\log\tfrac{q(y)}{p_{j}(y)} (A.10) = ∑ y ∈ 𝒱 q ⁡ ( y ) ​ log ⁡ q ⁡ ( y ) \displaystyle=\sum_{y\in\mathcal{V}}q(y)\log q(y) − ∑ y ∈ 𝒱 q ( y ) 𝔼 j [ log p j ( y ) ] , \displaystyle-\sum_{y\in\mathcal{V}}q(y)\,\mathbb{E}_{j}\!\left[\log p_{j}(y)\right],

Minimizing Eq. ( A.10 ) over q q subject to ∑ y q ⁡ ( y ) = 1 \sum_{y}q(y)=1 , we form the Lagrangian ℒ ⁡ ( q , λ ) \displaystyle\mathcal{L}(q,\lambda) = ∑ y q ⁡ ( y ) ​ log ⁡ q ⁡ ( y ) \displaystyle=\sum_{y}q(y)\log q(y) (A.11) − ∑ y q ( y ) 𝔼 j [ log p j ( y ) ] \displaystyle-\sum_{y}q(y)\,\mathbb{E}_{j}[\log p_{j}(y)] + λ ⁡ ( ∑ y q ⁡ ( y ) − 1 ) , \displaystyle+\lambda\!\left(\sum_{y}q(y)-1\right), where λ \lambda is the Lagrange multiplier for the normalization constraint. Setting ∂ ℒ / ∂ q ⁡ ( y ) = 0 \partial\mathcal{L}/\partial q(y)=0 yields the stationarity condition log ⁡ q ⁡ ( y ) + 1 − 𝔼 j ​ [ log ⁡ p j ​ ( y ) ] + λ = 0 , \log q(y)+1-\mathbb{E}_{j}\!\left[\log p_{j}(y)\right]+\lambda=0, (A.12) which gives q ⁡ ( y ) ∝ exp ⁡ ( 𝔼 j ​ [ log ⁡ p j ​ ( y ) ] ) q(y)\propto\exp\!\bigl(\mathbb{E}_{j}[\log p_{j}(y)]\bigr) . Enforcing the normalization constraint yields q ⋆ ​ ( y ) = exp ⁡ ( 𝔼 j ​ [ log ⁡ p j ​ ( y ) ] ) ∑ y ′ ∈ 𝒱 exp ⁡ ( 𝔼 j ​ [ log ⁡ p j ​ ( y ′ ) ] ) . q^{\star}(y)\;=\;\frac{\exp\!\bigl(\mathbb{E}_{j}[\log p_{j}(y)]\bigr)}{\sum_{y^{\prime}\in\mathcal{V}}\exp\!\bigl(\mathbb{E}_{j}[\log p_{j}(y^{\prime})]\bigr)}. (A.13) Equivalently, q ⋆ q^{\star} is the normalized weighted geometric mean of the teacher distributions: q ⋆ ​ ( y ) ∝ ∏ j p j ​ ( y ) μ x ​ ( j ) . q^{\star}(y)\;\propto\;\prod_{j}p_{j}(y)^{\mu_{x}(j)}. (A.14)

#### Interpretation

Reverse-KL direct OPSD replaces the per-token arithmetic average p ¯ \bar{p} from the forward-KL case (Eq. ( A.8 )) with a normalized geometric average of the self-conditioned teacher distributions. The two averages have very different behavior: the geometric mean q ⋆ ​ ( y ) q^{\star}(y) is large only when every p j ​ ( y ) p_{j}(y) is non-negligible, so reverse KL exhibits mode-seeking behavior, concentrating mass on tokens that all teachers find plausible. The arithmetic mean p ¯ ​ ( y ) \bar{p}(y) , in contrast, is large whenever any p j ​ ( y ) p_{j}(y) is large, exhibiting mode-covering behavior. Despite this difference, the overall conclusion is the same: changing the divergence merely changes the form of averaging, but the optimization target is still constructed from self-conditioned teacher distributions.

### A.4 Summary: Why Direct OPSD Fails in RLVR

Combining Appendices A.2 and A.3 , we obtain a unified characterization of direct OPSD at any prefix s i , t s_{i,t} .

#### Pointwise minimizers under different divergences.

Let { p j } j \{p_{j}\}_{j} denote the self-conditioned teacher distributions induced by verifier-approved trajectories τ j ∈ 𝒞 ⁡ ( x ) \tau_{j}\in\mathcal{C}(x) , weighted by the selection rule μ x \mu_{x} . Then the pointwise minimizer of direct OPSD takes the form q ⋆ ​ ( y ) = { ∑ j μ x ​ ( j ) ​ p j ​ ( y ) (forward KL), ∏ j p j ​ ( y ) μ x ​ ( j ) ∑ y ′ ∏ j p j ​ ( y ′ ) μ x ​ ( j ) (reverse KL), q^{\star}(y)\;=\;\begin{cases}\displaystyle\sum_{j}\mu_{x}(j)\,p_{j}(y)&\text{(forward KL),}\\[6.0pt] \displaystyle\frac{\prod_{j}p_{j}(y)^{\mu_{x}(j)}}{\sum_{y^{\prime}}\prod_{j}p_{j}(y^{\prime})^{\mu_{x}(j)}}&\text{(reverse KL),}\end{cases} (A.15) a weighted arithmetic average and a normalized weighted geometric average of the teachers, respectively.

#### Three structural problems.

Eq. ( A.15 ) exposes three problems that arise regardless of the choice of divergence:

The optimization target is an average, not a trajectory. Both minimizers are constructed by combining multiple self-conditioned teachers. Even if every individual p j p_{j} corresponded to a valid local continuation of τ j \tau_{j} , the average q ⋆ q^{\star} generally does not correspond to any single feasible trajectory. The student is trained to imitate a synthetic distribution that no verified rollout actually instantiates.

Disagreement between teachers carries information, but the loss discards it. The forward-KL decomposition in Eq. ( A.7 ) contains the disagreement term 𝒟 disagree ≥ 0 \mathcal{D}_{\mathrm{disagree}}\geq 0 , which measures how much the teachers { p j } \{p_{j}\} disagree about the next token at the current prefix. This quantity is large precisely at prefixes where different verified trajectories branch into different continuations—for example, a math prefix “ Solve 2 ​ x + 4 = 10 2x+4=10 , so ” where one verified trajectory continues with “ 2 ​ x = 6 2x=6 ” while another continues with “ subtract 4 from both sides ”. Such prefixes are arguably the most informative ones for the student: they mark decision points where multiple valid continuations exist. Yet because 𝒟 disagree \mathcal{D}_{\mathrm{disagree}} does not depend on q q , it contributes nothing to the gradient. As far as the loss is concerned, a prefix with strong teacher disagreement is treated identically to a prefix where all teachers agree. Direct OPSD therefore uses the disagreement signal only implicitly , through how it perturbs the average p ¯ \bar{p} , and never as a per-token quantity that could focus learning on these branch points.

The teacher is not a verifier-defined ground truth. Each p j p_{j} is constructed by conditioning the student on a single verifier-approved trajectory τ j \tau_{j} . As discussed in Section 5 , τ j \tau_{j} certifies only end-task success and may contain redundant, lucky, or even flawed reasoning. Treating p j p_{j} as a distillation target therefore propagates whatever artifacts τ j \tau_{j} contains into the student, weighted equally with genuinely informative predictions.

#### Implication.

The three problems above are not artifacts of a particular divergence choice or selection rule μ x \mu_{x} : they follow directly from the structural fact that direct OPSD uses self-conditioned teachers as distillation targets . Any objective that asks the student to move toward { p j } \{p_{j}\} in distribution will inherit some form of Eq. ( A.15 ), and hence inherit (P1)–(P3).

## Appendix B Experiment Details

### B.1 Training Details

Table 4 summarizes the training settings for each task. All experiments use Qwen3-8B (thinking disabled) as the base model. For mathematical reasoning (AIME 24&25), we randomly sample 4,000 problems from DAPO-Math-17k ( Yu et al., 2026 ) . For code generation (LCB v6), we randomly sample half of the unit tests per problem in LiveCodeBench v6 Jain et al. (2025) as our training set and reserve the other half for evaluation. For AppWorld Trivedi et al. (2024) , we train on the official training split. For WebShop Yao et al. (2022) , we randomly sample 2,400 tasks from the training split. All runs use 8 rollouts per prompt, a batch size of 8 prompts per update, and a learning rate of 1 × 10 − 6 1\times 10^{-6} . We conducted experiments using an 8-node cluster. Each node was equipped with 8 NVIDIA A100 80GB GPUs, 144-core AMD EPYC 7713 processors, and 960 GB of RAM, running Ubuntu as the operating system.

### B.2 OPSD Details

We follow the OPSD framework of Zhao et al. (2026) . In the original formulation, the teacher is conditioned on the ground-truth answer y ∗ y^{*} (e.g., an answer with reference chain-of-thought), and the training objective minimizes the per-token divergence between the privileged teacher and the student along the student’s own rollouts: ℒ OPSD ( θ ) = 𝔼 x , y ^ ∼ π θ ( ⋅ ∣ x ) [ 1 | y ^ | ∑ t = 1 | y ^ | D ( p T t ∥ p S t ) ] , \displaystyle\mathcal{L}_{\mathrm{OPSD}}(\theta)=\mathbb{E}_{x,\,\hat{y}\sim\pi_{\theta}(\cdot\mid x)}\left[\frac{1}{|\hat{y}|}\sum_{t=1}^{|\hat{y}|}D\big(p_{T}^{t}\,\big\|\,p_{S}^{t}\big)\right], where p T t = π θ ( ⋅ ∣ x , y ∗ , y ^ < t ) , \displaystyle\text{where}\quad p_{T}^{t}=\pi_{\theta}(\cdot\mid x,y^{*},\hat{y}_{<t}), p S t = π θ ( ⋅ ∣ x , y ^ < t ) . \displaystyle\phantom{\text{where}}\quad p_{S}^{t}=\pi_{\theta}(\cdot\mid x,\hat{y}_{<t}).

#### Demonstration Acquisition

The key requirement of OPSD is a ground-truth answer y ∗ y^{*} to serve as the teacher’s privileged context. How y ∗ y^{*} is obtained depends on whether ground-truth answers are available.

Math (DAPO-Math-17k). Ground-truth answers are available, so we directly use them as the privileged context y ∗ y^{*} to construct the self-conditioned teacher. To simulate varying teacher quality, we sample 500 questions and measure each external LLM’s accuracy; we then provide ground-truth to a corresponding fraction of training queries (matching that model’s coverage rate). Oracle provides ground truth to 100% of queries.

Code generation (LCB v6), AppWorld, and WebShop. No ground-truth solutions exist. We instead query each teacher LLM up to 3 times per query and retain any correct trajectory (verified by execution) as the demonstration τ \tau . Per-model coverage is reported in Table 5 .

#### Training Hyperparameters