Recursive Self-Improvement via On-Policy Distillation for Reasoning — part 3/4

Cleaned-text code points [30948, 46065). Bibliography omitted here; source.md includes it.

To explore the distinct roles of the two training objectives, we vary their coefficients separately on Qwen3-8B. The DCE weight λ G \lambda_{G} controls the strength of privileged next-token guidance, whereas the SRCL weight λ S \lambda_{S} controls the contribution of accepted concise rewrites. In each sweep, the other coefficient and all remaining training and evaluation settings are fixed. Table 3 reports the resulting task accuracy and output length for each setting.

Both sweeps identify a broad but non-monotonic operating region. Across the tested λ G \lambda_{G} range, Average accuracy varies only from 63.96% to 65.97%; λ G = 5 × 10 4 \lambda_{G}=5\times 10^{4} gives both the highest Average and the lowest mean length, although 1 × 10 4 1\times 10^{4} is stronger on AIME24 and HMMT25. For SRCL, λ S = 25 \lambda_{S}=25 likewise gives the highest Average, 65.97%, while the other settings remain within 2.57 points despite spanning a forty-fold range. These results indicate local robustness, not a monotonic or scale-independent optimum; the corresponding 4B sweep is reported in table 25 . They also clarify that the objectives are complementary: when DCE is removed, SRCL alone collapses to 2.36%/1,116 tokens at 8B and 0.76%/531 tokens at 4B. Concise self-refinement therefore improves the frontier only when paired with dynamic guidance that develops the underlying revision capability.

### 5.5 How Does the Guidance Objective Influence Performance?

To examine how the direction of distillation affects recursive improvement, we compare the best observed Qwen3-8B DCE+SRCL runs using Forward KL D KL ( q ∥ p ) D_{\mathrm{KL}}(q\|p) , Reverse KL D KL ( p ∥ q ) D_{\mathrm{KL}}(p\|q) , and JSD 1 2 D KL ( q ∥ m ) + 1 2 D KL ( p ∥ m ) \tfrac{1}{2}D_{\mathrm{KL}}(q\|m)+\tfrac{1}{2}D_{\mathrm{KL}}(p\|m) , where m = ( q + p ) / 2 m=(q+p)/2 , under the same evaluation protocol.

The results reflect the asymmetry of the three objectives. Forward KL weights discrepancies by the teacher distribution, preserving a strong signal for corrections that the student underweights.

Reverse KL instead weights the mismatch by the student distribution; corrections that the student rarely considers contribute less, favoring its existing modes over missing teacher-supported alternatives. JSD is symmetric and bounded, which can further weaken directional transfer when the two distributions differ substantially. Consistent with this interpretation, Forward KL reaches 65.97% with 17,561 tokens, whereas Reverse KL is 5.55 percentage points lower while using 6,187 more tokens. JSD reaches only 17.99% with 3,767 tokens, suggesting premature shortening rather than useful concision. Forward KL is therefore the most effective of the tested objectives for transferring privileged revision guidance.

\FloatBarrier

### 5.6 Does the Privileged Teacher Need to Co-Evolve?

To isolate the role of teacher refresh, we compare co-evolving and frozen gold-conditioned teachers under the same on-policy pipeline, both with and without SRCL. Figure 5 summarizes the accuracy–length comparison; complete trajectories and task-level results appear in appendices D , 6 and 13 .

Co-evolution improves both configurations. At 8B and 4B, DCE reaches 65.76% and 60.00%, versus 47.99% and 25.28% when frozen; DCE+SRCL reaches 65.97% and 61.88%, versus 41.04% and 27.29%. The shorter frozen runs incur large accuracy losses and regress after brief initial gains, indicating premature termination. Continual refresh instead sustains emerging revision behavior and keeps gold-conditioned guidance aligned with the evolving model.

\FloatBarrier

### 5.7 Which Representation of the Verified Solution Works Better?

Beyond teacher refresh, privileged-solution placement also affects signal quality. Assistant-side prefill places the solution in preceding model-generated context, whereas user-side instruction-last treats it as external material. Holding all else fixed, assistant-side prefill improves Average by 9.79, 6.39, and 14.66 points at 8B, 4B, and 1.7B and is 780 tokens shorter at 8B ( table 5 ). A matched 8B probe localizes its strongest effect near the start of Y 0 Y_{0} , whereas user-side effects persist later ( table 17 ), consistent with route initialization. Placement alone is insufficient: reference-last performs better under a frozen teacher ( table 16 ). The best result combines assistant-side conditioning with continual refresh; fig. 7 gives the exact prompt orderings.

\FloatBarrier

### 5.8 How Does the Method Perform under Tight Reasoning Budgets?

A practical question is whether the learned revision behavior remains useful under tight reasoning budgets. We therefore evaluate Qwen3-8B with total output budgets of 8K and 16K using three inference-only controls. Truncate directly cuts the response generated under the original 32K setting at the target budget. Cue first generates 7K tokens for an 8K budget (14K for 16K), appends a short instruction to continue reasoning, and generates the remainder within the same total cap. TTS adapts the test-time-scaling continuation procedure of Ghosal et al. (2025) : when an OPSD response ends early, it appends a Wait cue and continues until the output reaches the exact target length. These controls modify decoding only; model parameters remain fixed.

Table 6 shows that simply forcing more tokens is ineffective: at 8K, TTS lowers OPSD from 29.17% to 28.47% while nearly doubling output length, and at 16K it gains only 0.48 points while adding 11,275 tokens. Cue, by contrast, improves every matched DCE setting while slightly reducing mean output. At 8K, DCE rises from 24.38% to 31.04% and DCE+SRCL from 28.54% to 35.07%; at 16K, the corresponding gains are 48.68% to 55.76% and 52.01% to 57.64%. Thus the learned revision behavior remains useful under tight budgets, whereas length alone does not explain the gain. Complete TTS results across model sizes appear in table 23 .

### 5.9 How Sensitive Is Performance to Decoding Hyperparameters?

(a) Temperature sweep ( ρ = 1.0 \rho=1.0 )

(b) Repetition-penalty sweep ( T = 1.0 T=1.0 )

Finally, we test sensitivity to two common decoding choices under the non-thinking, 32K, Average@12 protocol. Table 7 shows that the default T = 1.0 T=1.0 gives the best Average among the tested temperatures (65.97%/17,561 tokens), while ρ = 1.08 \rho=1.08 is locally best among the tested repetition penalties (67.15%/17,173), gaining 1.18 points with 388 fewer tokens. We retain default decoding for the main comparisons and report the repetition-penalty result as a decoding-sensitivity analysis.

## 6 Conclusion

We presented Dynamic Co-Evolution (DCE), which makes privileged self-distillation recursive: every updated checkpoint becomes the next student and the next detached teacher, so revision learned in one round shapes the next round’s supervision. Self-Refined Concise Learning (SRCL) complements it with shorter, answer-verified rewrites of the model’s own responses; together, they deliver stronger and more token-efficient recursive self-improvement across model scales.

## Appendix A Does Recursive Improvement Scale to a Larger Model?

To explore whether recursive improvement extends to a larger model, we train Qwen3-14B with the same DCE+SRCL framework and track its early trajectory in table 8 . Average accuracy rises from 21.39% at initialization to 42.64% at step 20 and 66.39% at step 30, a total gain of 45.00 percentage points. The improvement spans all four benchmarks and is largest on AIME26, which rises by 50.28 percentage points, from 19.44% to 69.72%. Step 50 retains a comparable 65.21%, although mean output grows from 15,877 to 20,050 tokens. These results show strong round-to-round improvement at 14B while indicating that early stopping remains important for preserving the accuracy–length frontier.

## Appendix B Gemma-4-12B-IT Training Details and Additional Results

The cross-family comparison appears in table 2 . The exact model ID is google/gemma-4-12B-it . The matched DCE runs use the same 14,717-problem OPSD shard, assistant-side privileged conditioning, non-thinking student and teacher branches, and 200-step LoRA training with seed 42 (rank 128, alpha 256, global batch 16, and learning rate 2.5 × 10 − 6 2.5\times 10^{-6} ). Their only objective-level difference is λ S = 0 \lambda_{S}=0 for DCE and λ S = 25 \lambda_{S}=25 for DCE+SRCL, with λ G = 5 × 10 4 \lambda_{G}=5\times 10^{4} in both.

Placement also matters on Gemma-4-12B-IT. Assistant-side conditioning reaches 63.61% Average@12, compared with 55.76% for user-side reference-last and 54.31% for user-side instruction-last ( table 9 ). Both user-side variants attain their best observed result at step 10, whereas the assistant-side run continues improving through step 100.

## Appendix C Complete Qwen3-8B and Qwen3-4B Trajectories

Tables 11 and 12 report checkpoint-wise Average@12 accuracy and mean output length for Qwen3-8B and Qwen3-4B, complementing the learning curves in fig. 3 . Missing entries denote unevaluated checkpoints. The GRPO columns show the available trajectories, while table 1 reports the tuned baseline comparison.

SRCL without DCE collapses to short, mostly incorrect responses: Qwen3-8B reaches at most 2.36% with 1,116 tokens, and Qwen3-4B reaches 0.76% with 531 tokens. SRCL therefore acts as a concision objective only when paired with dynamic guidance.

\FloatBarrier

## Appendix D Teacher-Update Schedules

We isolate the effect of teacher refresh by comparing frozen, exponential-moving-average (EMA), periodic-snapshot, and fully dynamic teachers under the same Qwen3-8B setup. For EMA, the teacher after optimizer step t t is θ T ( t ) = d ​ θ T ( t − 1 ) + ( 1 − d ) ​ θ S ( t ) , \theta_{T}^{(t)}=d\,\theta_{T}^{(t-1)}+(1-d)\,\theta_{S}^{(t)}, (9) where a smaller decay d d follows the student more closely. Periodic refresh instead copies the current student into the teacher every m m optimizer steps and keeps the teacher fixed between updates. The frozen teacher is never refreshed, while the dynamic teacher is synchronized at every step. Table 13 summarizes task-level performance, while tables 14 and 15 give the complete Average@12 trajectories.

Every refresh strategy substantially improves over the frozen teacher. EMA d = 0.3 d=0.3 reaches the highest EMA accuracy at 65.35%, while d = 0.7 d=0.7 offers the better accuracy–length balance at 65.00% with 18,412 tokens. Periodic snapshots are weaker, reaching at most 63.47%. Updating the teacher every step performs best overall at 65.97% with 17,561 tokens. Because each schedule is represented by one training trajectory, small differences should not be overinterpreted.

\FloatBarrier

## Appendix E Effect of Privileged-Context Placement

The placement ablation separates teacher refresh from the representation of the verified solution. Table 16 reports the complete Qwen3-8B comparison across the three prompt formats illustrated in fig. 7 . Every dynamic variant outperforms its frozen counterpart. With a dynamic teacher, assistant-side prefill is strongest, reference-last is intermediate, and instruction-last is weakest. Under a frozen teacher, reference-last performs best but remains far below the dynamic variants.

After generating Y 0 Y_{0} without g g , we hold it fixed. At position t t , both branches score the same Y 0 , < t Y_{0,<t} ; only the privileged branch also receives g g , and neither observes future tokens.

Table 17 shows a front-loaded, not uniformly stronger, intervention. At positions 1–32, assistant-side has the highest KL/NLL, indicating the largest distribution shift and less probability on the observed Y 0 Y_{0} token. After position 32, both metrics are lowest, indicating agreement with the fixed continuation rather than correctness. This pattern is consistent with route initialization, while the frozen reversal in table 16 shows that placement alone is insufficient.

#### Evolution of the privileged signal.

We next track the same diagnostic over training. At each checkpoint, KL compares the verified-solution-conditioned teacher distribution q T q_{T} with the no-GOLD student distribution p S p_{S} on the same fixed correct trajectories. KL records how strongly the privileged context changes the prediction, but not whether that change favors the observed correct continuation. We therefore also report Δ ​ NLL = NLL no-GOLD − NLL with-GOLD , \Delta\mathrm{NLL}=\mathrm{NLL}_{\text{no-GOLD}}-\mathrm{NLL}_{\text{with-GOLD}}, (10) where a positive value means that privileged conditioning assigns greater likelihood to the fixed correct continuation.

The two metrics separate intervention strength from direction: KL asks how much GOLD changes the prediction, while Δ \Delta NLL asks whether that change favors the recorded correct continuation. For assistant-side conditioning, both signals increase throughout training: KL rises monotonically from 0.1303 to 0.4371, and Δ \Delta NLL rises from + 0.0414 +0.0414 to + 0.6474 +0.6474 . Assistant-side also has the largest Δ \Delta NLL at every checkpoint, with its advantage widening later in training. The two user-side runs strengthen through step 50 but then saturate or regress. From step 50 to 100, Δ \Delta NLL increases from + 0.3729 +0.3729 to + 0.4139 +0.4139 for assistant-side, but falls from + 0.2209 +0.2209 to + 0.0266 +0.0266 for reference-last and from + 0.1492 +0.1492 to + 0.0026 +0.0026 for instruction-last. At step 200, the assistant-side improvement exceeds reference-last and instruction-last by 0.6702 and 0.5484, respectively. Thus, the user-side contexts can continue to alter the output distribution without reliably increasing likelihood on the recorded correct path.

These curves show three training trajectories progressively separating; they do not by themselves identify prompt position as the cause because each column comes from a separately trained model. Nor is larger KL synonymous with higher benchmark accuracy: assistant-side accuracy peaks before its KL does. Finally, Δ \Delta NLL measures alignment with the fixed correct trajectories used in this probe, not final-answer accuracy or support for every valid derivation.

\FloatBarrier

## Appendix F Qwen3-1.7B Training Stability

At 1.7B, both recursive variants improve rapidly but become unstable later in training: DCE peaks at step 30, while DCE+SRCL peaks at step 50 before declining. Table 19 reports the complete trajectory and highlights the importance of early stopping at this scale.

\FloatBarrier

## Appendix G Fixed-Trace Probe Values

Figure 4 summarizes endpoint EOS and reflection-cue probabilities on fixed cohorts from AIME24, AIME25, and AIME26. The cohorts contain 227, 212, and 245 incorrect responses and 879, 805, and 583 observed revision events, respectively. At each event, r t r_{t} is the token that appears when the stored trajectory begins to revise, such as Wait . Table 20 reports the complete AIME26 values, including EOS probability at the same pre-reflection positions.

\FloatBarrier

## Appendix H EOS-Penalty Ablation

#### Objective.