Recursive Self-Improvement via On-Policy Distillation for Reasoning — part 2/4

Cleaned-text code points [15410, 30948). Bibliography omitted here; source.md includes it.

DCE alternates on-policy generation with updates to both the student and its privileged teacher. Let θ k \theta_{k} denote the model parameters at training round k k , with θ 0 \theta_{0} representing the initial checkpoint. Given a problem x x , the current model samples an on-policy response y ( k ) ∼ p θ k ( ⋅ ∣ x ) . y^{(k)}\sim p_{\theta_{k}}(\cdot\mid x). (2) From this response, we retain a sequence y ~ ( k ) = ( y ~ 1 ( k ) , … , y ~ m ( k ) ) \widetilde{y}^{(k)}=(\widetilde{y}^{(k)}_{1},\ldots,\widetilde{y}^{(k)}_{m}) of m m tokens on which the guidance objective is computed. At each token position t ∈ { 1 , … , m } t\in\{1,\ldots,m\} , the student and privileged teacher score the same response prefix y ~ < t ( k ) \widetilde{y}^{(k)}_{<t} . The student conditions on the problem x x and this prefix, whereas the privileged teacher additionally receives the ground-truth solution g g . Its prompt places the problem x x and task instruction in the user turn, followed in a single assistant turn by g g , a transition instruction τ \tau asking the model to solve the problem using its own approach, and finally y ~ < t ( k ) \widetilde{y}^{(k)}_{<t} . We denote this privileged context by 𝒞 t prev ​ ( x , g , τ , y ~ < t ( k ) ) \mathcal{C}_{t}^{\mathrm{prev}}(x,g,\tau,\widetilde{y}^{(k)}_{<t}) , i.e. the concatenation of all privileged inputs up to position t t . This ordering is deliberate: placing g g in the assistant turn lets the model condition on it as part of its own reasoning rather than as external user-provided material. The exact templates and alternative orderings are given in fig. 7 ; their performance is compared in section 5.7 .

We define the student and privileged-teacher next-token distributions at round k k and position t t as p k , t \displaystyle p_{k,t} : = p θ k ( ⋅ ∣ x , y ~ ( k ) < t ) , \displaystyle:=p_{\theta_{k}}\!\left(\cdot\mid x,\widetilde{y}^{(k)}_{<t}\right), (3) q k , t \displaystyle q_{k,t} : = stopgrad [ p θ k ( ⋅ ∣ 𝒞 t prev ( x , g , τ , y ~ ( k ) < t ) ) ] . \displaystyle:=\operatorname{stopgrad}\!\left[p_{\theta_{k}}\!\left(\cdot\mid\mathcal{C}_{t}^{\mathrm{prev}}(x,g,\tau,\widetilde{y}^{(k)}_{<t})\right)\right]. Here p k , t p_{k,t} is the student distribution, conditioned only on the problem, and q k , t q_{k,t} is the privileged-teacher distribution, which additionally sees the ground-truth solution. Both are produced by the same current checkpoint θ k \theta_{k} , but gradients are stopped through q k , t q_{k,t} . In contrast, standard OPSD differs in two ways: it presents the ground-truth solution as reference material in the user turn rather than as previously generated reasoning in the assistant turn, and it keeps the privileged teacher frozen at the initial checkpoint θ 0 \theta_{0} throughout training. Let Δ ⁡ ( q , p ) \Delta(q,p) denote a divergence measuring the mismatch between the teacher and student next-token distributions. DCE minimizes ℒ G ( k ) = 𝔼 ( x , g ) , y ( k ) ∼ p θ k ( ⋅ ∣ x ) [ 1 m ∑ t = 1 m Δ ( q k , t , p k , t ) ] . \mathcal{L}_{G}^{(k)}=\mathbb{E}_{(x,g),\,y^{(k)}\sim p_{\theta_{k}}(\cdot\mid x)}\!\left[\frac{1}{m}\sum_{t=1}^{m}\Delta\!\left(q_{k,t},p_{k,t}\right)\right]. (4) After optimizing this objective at round k k , the resulting parameters θ k + 1 \theta_{k+1} initialize both the student and privileged teacher for the next round. Consequently, revision behavior acquired during one round can become part of the gold-conditioned supervision provided in subsequent rounds, yielding recursive co-evolution. We use Forward KL for Δ \Delta in our main experiments and compare Reverse KL and JSD in section 5.5 .

### 3.2 Self-Refined Concise Learning

DCE improves reasoning by strengthening the model’s ability to revisit and revise its own trajectories. However, stronger revision behavior can also increase inference cost: the model may perform repeated checks, explore unnecessary branches, or continue reasoning after it has already reached a correct solution. DCE alone does not explicitly encourage the model to preserve useful reasoning while removing these redundant steps. We therefore introduce Self-Refined Concise Learning (SRCL) , which complements DCE by training the model on shorter, verified refinements of its own on-policy responses.

At training round k k , let ℬ = { ( x i , g i ) } \mathcal{B}=\{(x_{i},g_{i})\} denote a minibatch of problems x i x_{i} and their verified solutions g i g_{i} . For each example i i , the current checkpoint θ k \theta_{k} generates an on-policy response y i ( k ) y_{i}^{(k)} . SRCL then asks the same checkpoint to rewrite this response without access to g i g_{i} :

c i ( k ) = R θ k ​ ( x i , y i ( k ) ) , c_{i}^{(k)}=R_{\theta_{k}}\!\left(x_{i},y_{i}^{(k)}\right), (5) where R θ k R_{\theta_{k}} denotes generation under a refinement prompt that requests a direct, self-contained solution with unnecessary detours and reflection removed. We retain the refinement c i ( k ) c_{i}^{(k)} only if it is shorter than the original response y i ( k ) y_{i}^{(k)} , terminates naturally, satisfies structural validity requirements, and produces the correct answer. Let A i ( k ) ∈ { 0 , 1 } A_{i}^{(k)}\in\{0,1\} indicate whether c i ( k ) c_{i}^{(k)} passes all of these criteria; the exact acceptance rules are given in appendix M .

Each accepted refinement becomes a token-level supervised target. SRCL minimizes the autoregressive cross-entropy over all accepted tokens: ℒ SRCL ( k ) = − 1 N k ∑ i ∈ ℬ A i ( k ) ∑ t = 1 | c i ( k ) | log p θ k ( c i , t ( k ) ∣ x i , c i , < t ( k ) ) , N k = ∑ i ∈ ℬ A i ( k ) | c i ( k ) | . \mathcal{L}_{\mathrm{SRCL}}^{(k)}=-\frac{1}{N_{k}}\sum_{i\in\mathcal{B}}A_{i}^{(k)}\sum_{t=1}^{|c_{i}^{(k)}|}\log p_{\theta_{k}}\!\left(c_{i,t}^{(k)}\mid x_{i},c_{i,<t}^{(k)}\right),\qquad N_{k}=\sum_{i\in\mathcal{B}}A_{i}^{(k)}|c_{i}^{(k)}|. (6) Here, i i indexes examples in the minibatch and t t indexes tokens within an accepted refinement. If no refinement is accepted in a minibatch, such that N k = 0 N_{k}=0 , we omit the SRCL loss and update the model using DCE alone. Importantly, the verified solution g i g_{i} is used only to determine whether a refinement is accepted; it is never provided to the model when generating c i ( k ) c_{i}^{(k)} , preventing ground-truth leakage into the learned rewriting behavior. The exact refinement prompt, structural acceptance criteria, and answer-verifier implementation are provided in appendices M and N .

### 3.3 Joint Optimization

DCE and SRCL provide complementary supervision from the same on-policy response. DCE trains the model to follow the privileged teacher at prefixes encountered along the current rollout, whereas SRCL trains on shorter, verified refinements derived from that rollout. We combine the two objectives as ℒ DCE + SRCL ( k ) = λ G ​ ℒ G ( k ) ⏟ DCE guidance + λ S ​ ℒ SRCL ( k ) ⏟ SRCL target . \mathcal{L}_{\mathrm{DCE+SRCL}}^{(k)}=\underbrace{\lambda_{G}\mathcal{L}_{G}^{(k)}}_{\text{DCE guidance}}+\underbrace{\lambda_{S}\mathcal{L}_{\mathrm{SRCL}}^{(k)}}_{\text{SRCL target}}. (7) Here, ℒ G ( k ) \mathcal{L}_{G}^{(k)} is the dynamic guidance loss defined in eq. 4 , and ℒ SRCL ( k ) \mathcal{L}_{\mathrm{SRCL}}^{(k)} is the self-refinement loss defined in eq. 6 . The coefficients λ G \lambda_{G} and λ S \lambda_{S} control the relative contribution of the two objectives; their empirical sensitivity is studied in section 5.4 . The complete recursive procedure is summarized in algorithm 1 .

## 4 Experimental Setup

### 4.1 Models, Data, and Metrics

We adopt Qwen3-1.7B, Qwen3-4B, Qwen3-8B, and Qwen3-14B ( Qwen Team, 2025 ) as our primary model family, and Gemma-4-12B-IT in the cross-family study in section 5.2 . Both the problem-only student and gold-conditioned privileged teacher operate in non-thinking mode. All methods use the same 14,717 problems drawn from the OpenThoughts mathematical-reasoning data adopted by OPSD ( Guha et al., 2025 ; Zhao et al., 2026 ) . DCE constructs student and privileged views of each problem during training. Complete configurations are reported in table 24 .

Evaluation uses all 30 problems from each of AIME 2024, AIME 2025, AIME 2026 ( Mathematical Association of America, 2026 ) , and HMMT February 2025 ( Harvard–MIT Mathematics Tournament, 2026 ) . We sample 12 responses per problem with a 32K generation cap, temperature 1.0, top- p = 0.8 p=0.8 , top- k = − 1 k=-1 , and repetition penalty 1.0.

### 4.2 Baselines and Controls

We compare against four training baselines: Base , SFT , GRPO , and OPSD . Base is the original model evaluated in non-thinking mode. SFT trains on the problem–solution pairs ( x , g ) (x,g) using autoregressive cross-entropy, while GRPO optimizes an outcome reward based on final-answer correctness. OPSD is our closest training baseline: it distills next-token predictions from a frozen, gold-conditioned privileged teacher initialized from the same base checkpoint.

To separate the effect of improved training from that of simply allocating more inference tokens, we additionally include OPSD-TTS as an inference-time control. Following the extended-thinking intervention of Ghosal et al. (2025) , when an OPSD response terminates before the target budget, a Wait cue resumes generation until the response reaches an exact 8K or 16K token budget. OPSD-TTS does not modify the model parameters and therefore tests whether longer generation alone can account for the gains of DCE+SRCL.

### 4.3 Training Protocol

Across the Qwen3 DCE experiments, we train with AdamW ( Loshchilov and Hutter, 2019 ) in bfloat16 using a learning rate of 5 × 10 − 6 5\times 10^{-6} and rank-128 LoRA ( Hu et al., 2022 ) . Each training example produces a single on-policy response. DCE computes privileged-teacher guidance over the retained response tokens, while SRCL greedily rewrites the same response and trains only on accepted targets that are shorter, reflection-free, and answer-correct.

We set the DCE loss weight to λ DCE = 5 × 10 4 \lambda_{\mathrm{DCE}}=5\times 10^{4} for all Qwen3 models. For SRCL, we use λ SRCL = 12.5 , 25 , 35 , \lambda_{\mathrm{SRCL}}=12.5,25,35, and 25 25 for Qwen3-14B, 8B, 4B, and 1.7B, respectively. Sensitivity to both coefficients is analyzed in section 5.4 . We evaluate checkpoints at training steps { 10 , 20 , 30 , 50 , 100 , 150 , 200 } \{10,20,30,50,100,150,200\} when available.

Baseline-specific SFT and GRPO hyperparameters are reported in table 24 , and checkpoint-level learning trajectories are provided in figs. 3 and C . The Gemma transfer configuration is reported separately in appendix B .

## 5 Results

### 5.1 Main Experimental Results

Table 1 summarizes the main results. DCE substantially improves over the training baselines across model scales. For Qwen3-8B, DCE+SRCL reaches 65.97% Average@12, compared with 30.35% for OPSD and 20.28% for GRPO. The gains remain substantial with Qwen3-4B, where DCE+SRCL reaches 61.88%, compared with 22.85% for OPSD and 18.47% for GRPO. DCE alone achieves similar accuracy with Qwen3-8B (65.76%) and reaches 60.00% with Qwen3-4B, indicating that the primary accuracy gains come from dynamic co-evolution.

SRCL improves the accuracy–length tradeoff at these scales. With Qwen3-8B, adding SRCL reduces mean output length from 19,046 to 17,561 tokens while maintaining comparable Average@12 accuracy (65.76% versus 65.97%). With Qwen3-4B, SRCL both improves accuracy from 60.00% to 61.88% and reduces mean output length from 19,360 to 17,265 tokens, a 10.82% reduction. This effect is not uniform at the smallest scale: with Qwen3-1.7B, DCE+SRCL improves Average@12 from 23.33% to 26.88%, but also increases mean output length from 12,600 to 19,504 tokens.

Longer inference alone does not explain the gains. Forcing OPSD to generate exactly 16K tokens yields only 30.76%, 22.99%, and 9.24% Average@12 for Qwen3-8B, Qwen3-4B, and Qwen3-1.7B, respectively. The matched training trajectories in fig. 3 further separate the effect of SRCL from checkpoint selection: at step 100 with Qwen3-8B, DCE+SRCL reaches 65.97% with 17,561 mean tokens, compared with 64.93% and 19,036 tokens for DCE at the same step. With Qwen3-4B, DCE peaks earlier and declines after step 50, whereas DCE+SRCL remains near 61% through step 200. Complete checkpoint trajectories and budget controls are provided in appendices C , F , 5.8 and 23 .

\FloatBarrier

### 5.2 Does Recursive Improvement Transfer Beyond Qwen3?

To test whether recursive improvement transfers across model families, we apply our framework to Gemma-4-12B-IT. As shown in table 2 , DCE and DCE+SRCL reach 62.01% and 63.61% Average@12, respectively, outperforming all non-DCE comparisons. SRCL adds 1.60 percentage points while reducing mean output length from 8,812 to 8,526 tokens. Across the saved trajectory, DCE+SRCL attains higher accuracy at five of seven checkpoints, ties once, and is both more accurate and shorter than DCE at four checkpoints. The framework therefore transfers beyond Qwen3. Detailed configurations and complete training trajectories appear in appendix B .

\FloatBarrier

### 5.3 What Happens During Recursive Self-Improvement?

To characterize recursive self-improvement, we evaluate checkpoints from a Qwen3-8B DCE+SRCL run on a fixed set of incorrect trajectories. Holding the trajectories constant isolates two next-token behaviors: whether the model terminates after a wrong solution, and whether it predicts the reflection cue observed when the solution begins to revise.

For a stored response y i − y_{i}^{-} with its terminal EOS removed, the Student scores [ x i ; y i − ] [x_{i};y_{i}^{-}] , while the Teacher scores the same response with the verified solution prepended. At an observed revision point t t , we instead score the preceding prefix and its actual next cue r i , t r_{i,t} , such as Wait . Denote the resulting endpoint and revision contexts by c end c^{\mathrm{end}} and c ref c^{\mathrm{ref}} . For branch b ∈ { S , T } b\in\{S,T\} , the two probes are P b ( k ) ​ ( EOS ) = 1 | ℰ | ​ ∑ i ∈ ℰ p θ k ​ ( EOS ∣ c i , b end ) , P b ( k ) ​ ( r ) = 1 | 𝒲 | ​ ∑ ( i , t ) ∈ 𝒲 p θ k ​ ( r i , t ∣ c i , t , b ref ) . P_{b}^{(k)}(\mathrm{EOS})=\frac{1}{|\mathcal{E}|}\sum_{i\in\mathcal{E}}p_{\theta_{k}}(\mathrm{EOS}\mid c^{\mathrm{end}}_{i,b}),\qquad P_{b}^{(k)}(r)=\frac{1}{|\mathcal{W}|}\sum_{(i,t)\in\mathcal{W}}p_{\theta_{k}}(r_{i,t}\mid c^{\mathrm{ref}}_{i,t,b}). (8) Both values come directly from the full next-token distribution. The reflection probe scores the cue actually present in the stored trajectory rather than summing over a hand-built lexicon.

Figure 4 shows the same qualitative transition on all three benchmarks. Macro-averaged across the cohorts, Student/Teacher endpoint p ⁡ ( EOS ) p(\mathrm{EOS}) falls from 92.6%/90.4% at initialization to 26.3%/41.3% at step 200. Over the same interval, p ⁡ ( r t ) p(r_{t}) rises from 30.1%/32.8% to 77.0%/77.4%, with most of the increase occurring by steps 50–100. Recursive training therefore changes not only the student policy but also the privileged branch used to supervise the next round: both become less likely to stop after a wrong solution and more likely to support the observed reflection cue. This shared shift suggests that co-evolution transfers emerging revision behavior into the privileged teacher, complementing the frozen-teacher evidence in fig. 1 ; complete AIME26 probe values are reported in table 20 .

\FloatBarrier

### 5.4 How Do Different Loss Weights Affect Performance?