Recursive Self-Improvement via On-Policy Distillation for Reasoning — part 1/4

Cleaned-text code points [0, 15410). Bibliography omitted here; source.md includes it.

\microtypesetup expansion=false

# Recursive Self-Improvement via On-Policy Distillation for Reasoning

###### Abstract

On-policy distillation (OPD) trains a student model by having it generate trajectories, then matching its next-token predictions with an external teacher’s next-token predictions. This provides a dense, token-level supervision to the student. On-policy self-distillation (OPSD) eliminates the need for the external teacher. Specifically, a second frozen copy of the student model, now given the ground truth in its context, serves as the teacher. The student model only receives the problem and learns to mimic the privileged teacher model, while the teacher remains frozen throughout training. Previous work showed that freezing the teacher is useful for training stability, but we argue that this can prevent the teacher from incorporating the improvements learned by the student during training. Our primary contribution is to address this limitation with a recursive framework built around two complementary components. First, we let the privileged teacher to co-evolve with the student so that revision learned in one round can guide the next, a process we refer to as Dynamic Co-Evolution (DCE) . Second, because stronger revision can also make responses too verbose and self-critical, we additionally train on shorter, verified rewrites of the model’s own on-policy responses. We call this complementary objective Self-Refined Concise Learning (SRCL) . Overall, our comprehensive evaluations show that DCE+SRCL outperforms OPSD across multiple model scales and four competition-level mathematics benchmarks. Specifically, on Qwen3-8B, DCE+SRCL reaches 65.97% Average@12, outperforming OPSD by 35.62 percentage points while reducing mean output length by 7.80% relative to DCE alone.

## 1 Introduction

Post-training has played a central role in recent advances in LLM reasoning. A prominent approach is reinforcement learning with verifiable rewards (RLVR), which uses an automatic checker to assign an outcome reward to a completed solution ( Shao et al., 2024 ; Yu et al., 2025 ) . Because correctness can be verified without annotating every intermediate step, RLVR scales without step-level supervision and can elicit longer derivations, intermediate verification, backtracking, and self-correction ( Guo et al., 2025 ) . However, outcome rewards provide only coarse, sequence-level supervision: they indicate whether the final answer is correct, but not where the reasoning went wrong or how it should be revised. Process supervision provides more local feedback, but generally requires step-level annotations or a separately trained verifier ( Lightman et al., 2024 ; Zhang et al., 2025b ) .

Between sparse outcome rewards and costly process supervision, on-policy distillation (OPD) offers a third route: dense token-level targets on trajectories sampled from the model being trained ( Agarwal et al., 2024 ; Lu and Lab, 2025 ) . This model serves as the student, while a separate, stronger teacher provides next-token logits for each generated prefix. These logits provide a richer token-level training signal than the student’s own predictions. However, standard OPD still requires a capable external teacher, whose guidance is not conditioned on a verified solution for the current problem. The student is therefore trained to match the teacher’s token-level predictions, potentially inheriting its errors as well as its strengths. On-policy self-distillation (OPSD) ( Zhao et al., 2026 ) addresses these limitations using two roles initialized from the same base checkpoint. During training, the student model is given only the problem and generates an on-policy response. A copy of the base checkpoint is additionally given the ground-truth solution and serves as the privileged teacher. At each prefix of the student’s response, the teacher receives the problem, the ground-truth solution, and the same generated prefix, then supplies next-token logits. OPSD minimizes the divergence between the student and teacher predictions, transferring gold-conditioned guidance to the student. To stabilize training, the teacher remains frozen in Zhao et al. (2026) .

Does access to the ground-truth solution provide informative guidance at every student-generated prefix? We probe this question on fixed incorrect Qwen3-8B trajectories from AIME 2024, AIME 2025, and AIME 2026 ( Mathematical Association of America, 2026 ) , comparing evolving student checkpoints with the frozen, gold-conditioned teacher on the same generated prefixes. Despite seeing the verified solution, the frozen teacher assigns an average probability of 94.5% to EOS at incorrect response endpoints but only 22.0% to observed reflection cues such as Wait , a common reflection marker in reasoning models ( Wang et al., 2025 ) ( fig. 1 ), indicating limited guidance from the teacher on how to recover from incorrect reasoning. Meanwhile, the student becomes substantially more likely to reflect during training, creating a growing mismatch between the evolving model and its fixed supervisor.

This mismatch exposes a limitation of freezing the privileged teacher. Revision behaviors such as self-verification, backtracking, and error correction can strengthen during post-training ( Guo et al., 2025 ; Zhu et al., 2025 ) , but a teacher fixed at the initial checkpoint cannot acquire these emerging capabilities. Thus, even when conditioned on the verified solution, its guidance may become increasingly misaligned with the student trajectories encountered later in training. This observation motivates a privileged teacher that evolves with the student.

We therefore introduce Dynamic Co-Evolution (DCE), in which the privileged teacher evolves alongside the student. After each update, the resulting checkpoint initializes both the next student and a detached, gold-conditioned privileged teacher. Revision behavior acquired in one round can therefore improve the supervision provided in the next, creating a recursive self-improvement process. However, strengthening revision introduces a second challenge. More frequent checking, and backtracking can improve recovery from mistakes while also making reasoning unnecessarily long. We therefore pair DCE with Self-Refined Concise Learning (SRCL) , which trains on shorter, answer-verified rewrites of the same on-policy responses. DCE improves the model’s ability to revise its reasoning, while SRCL encourages it to retain that capability without unnecessary token cost.

Our contributions can be summarized as follows: • We introduce Dynamic Co-Evolution (DCE) , a recursive on-policy self-distillation framework in which each updated checkpoint initializes both the next student and a detached, gold-conditioned privileged teacher. Across four competition-level mathematical reasoning benchmarks, DCE improves Average@12 accuracy over OPSD by 35.41, 37.15, and 12.98 percentage points at 8B, 4B, and 1.7B, respectively.

• We introduce Self-Refined Concise Learning (SRCL) to control the reasoning cost that can accompany stronger revision. SRCL trains on accepted, shorter self-refinements of the same on-policy responses. Under our lowest tested 8K evaluation budget, DCE+SRCL achieves 35.07% Average@12 accuracy with 7,500 mean output tokens per response, outperforming the OPSD test-time-scaling control by 6.60 percentage points while generating approximately 692 fewer tokens per response.

• We show that the privileged teacher learns to provide stronger revision guidance as it evolves with the student. Matched-budget controls show that longer generation alone does not explain the gains. Updating the teacher every round also outperforms frozen, EMA, and periodic alternatives. Fixed-trace probes corroborate the teacher’s improved revision behavior across three AIME cohorts. Its endpoint EOS probability falls from 90.4% to 41.3%, while its probability on observed reflection tokens rises from 32.8% to 77.4%.

## 2 Related Work

### 2.1 Reflection and Verification in Reasoning Post-Training

Outcome-supervised post-training can elicit reflection-like behaviors without step-level labels. GRPO removes the learned critic used by PPO and estimates advantages from relative rewards within a sampled group, making outcome-based training practical for mathematical reasoning ( Shao et al., 2024 ) . DAPO introduces clip-higher, dynamic sampling, and token-level policy loss to improve the stability and efficiency of this recipe at scale ( Yu et al., 2025 ) . DeepSeek-R1 further demonstrates that outcome-based training alone can produce rechecking, backtracking, and “aha moments” without explicit reflection supervision ( Guo et al., 2025 ) . Parallel efforts extend RL-based post-training beyond closed-form math to policy-grounded content moderation ( Firooz et al., 2025 ) , open-ended environments requiring generalization without fixed-answer verification ( Yin et al., 2026a ) , and non-verifiable tasks balancing objective reasoning gains with subjective alignment ( Yin and Shi, 2026 ) . Despite this progress, outcome-level rewards score completed responses as a whole and do not supervise the local transition where the model identifies and repairs a specific error.

A closer look suggests that these reflection-like behaviors may not be newly learned through training. R1-Zero reproductions find similar behaviors already present in some base models and attribute part of the GRPO effect to a length bias toward longer, often incorrect, responses ( Liu et al., 2025 ) . Across ten base models, response length and verification behavior do not reliably emerge together ( Zeng et al., 2025 ) . Activation-space analysis further reveals a latent, though rare, capacity for reflection that exists before any RLVR ( Zhu et al., 2025 ) . Together, these results suggest that outcome-based post-training amplifies a pre-existing behavioral prior rather than teaching the model how to recover from a particular wrong prefix.

Even when reflection does appear in generation, it does not reliably correct errors. Without external feedback, prompting a model to revise its own answer can reduce accuracy ( Huang et al., 2024 ) , and hidden-state probes reveal correctness signals that the model’s generation does not always exploit ( Zhang et al., 2025a ; Lee et al., 2025 ) . The gap between latent awareness and effective revision motivates a different form of supervision: DCE provides dense, token-level guidance on the model’s own incorrect reasoning, directly training the transition from error recognition to successful revision.

### 2.2 On-Policy and Privileged Self-Distillation

OPD trains on responses sampled from the model being optimized, while a teacher supplies next-token supervision along those same responses ( Agarwal et al., 2024 ; Lu and Lab, 2025 ) . OPSD removes the external teacher by assigning a frozen copy of the initial checkpoint to the privileged role: the student sees only the problem, whereas the teacher also receives a verified solution ( Zhao et al., 2026 ) . Follow-up methods such as RLSD and RLCSD combine this signal with reinforcement learning ( Yang et al., 2026 ; Pan et al., 2026 ) .

Formally, let p θ p_{\theta} denote the language model with parameters θ \theta . The student conditions on a problem x x and generates a response y ∼ p θ ( ⋅ ∣ x ) y\sim p_{\theta}(\cdot\mid x) . The privileged teacher scores each prefix y < t y_{<t} after additionally receiving a verified solution g g and a transition instruction τ \tau , forming the privileged context 𝒞 t ref ​ ( x , g , τ , y < t ) \mathcal{C}_{t}^{\mathrm{ref}}(x,g,\tau,y_{<t}) . Let q q and p p denote the resulting teacher and student next-token distributions, respectively, and let Δ ⁡ ( q , p ) \Delta(q,p) denote a divergence between them. Our main experiments use Forward KL, Δ ( q , p ) = D KL ( q ∥ p ) = ∑ v q ( v ) log q ⁡ ( v ) p ⁡ ( v ) \Delta(q,p)=D_{\mathrm{KL}}(q\|p)=\sum_{v}q(v)\log\frac{q(v)}{p(v)} ; alternatives are compared in section 5.5 . Standard OPSD minimizes ℒ fixed ( θ ) = 𝔼 ( x , g ) , y [ 1 | y | ∑ t = 1 | y | Δ ( p θ 0 ( ⋅ ∣ 𝒞 t ref ( x , g , τ , y < t ) ) , p θ ( ⋅ ∣ x , y < t ) ) ] , \mathcal{L}_{\mathrm{fixed}}(\theta)=\mathbb{E}_{(x,g),y}\!\left[\frac{1}{|y|}\sum_{t=1}^{|y|}\Delta\!\left(p_{\theta_{0}}\!\left(\cdot\mid\mathcal{C}_{t}^{\mathrm{ref}}(x,g,\tau,y_{<t})\right),p_{\theta}(\cdot\mid x,y_{<t})\right)\right], (1) where the teacher remains fixed at θ 0 \theta_{0} and receives no gradients, even as the student and its generated responses change throughout training.

### 2.3 Self-Refinement and Efficient Reasoning

Self-training turns a model’s own generations into new training data. STaR iteratively trains on rationales that yield correct answers and uses answer-conditioned rationalization to recover additional examples, while ReST generates, filters, and reuses model samples as offline data ( Zelikman et al., 2022 ; Gülçehre et al., 2023 ) . In alignment, SAO generates its own prompts, responses, and preferences without external annotation ( Yin et al., 2026b ) , whereas PIKA uses an external generator and reward model to construct synthetic training data ( Yin et al., 2025 ) . Reflexion instead stores verbal feedback to guide later attempts without updating model parameters ( Shinn et al., 2023 ) , and subsequent work links verification, backtracking, and subgoal construction to successful self-improvement ( Gandhi et al., 2025 ) . SRCL follows this filtered self-training perspective but serves a different purpose: the current checkpoint rewrites its own on-policy response without seeing the verified solution, and training retains only rewrites that are shorter, naturally terminated, structurally valid, and answer-correct. SRCL therefore teaches concise successful solutions, complementing DCE’s guidance on how to revise the original response.

Test-time scaling improves accuracy by allocating more computation during inference. Repeated sampling, search, and compute-aware decoding trade additional inference compute for stronger performance ( Brown et al., 2024 ; Snell et al., 2025 ; Wu et al., 2025 ) , while input-adaptive methods allocate that compute according to estimated problem difficulty ( Damani et al., 2025 ) . Budget forcing specifically uses continuation cues to prevent early termination and extend a response to a prescribed budget ( Muennighoff et al., 2025 ) . SRCL instead changes the model during post-training so that a single rollout can preserve correct reasoning with fewer tokens. Our matched-budget controls therefore test whether the gains arise merely from longer generation or from the learned behavior.

## 3 Method

To address the limitations of a frozen privileged teacher while controlling the cost of increasingly long reasoning, our framework combines two complementary training objectives. Dynamic Co-Evolution (DCE) distills gold-conditioned guidance along the current student’s rollout and refreshes the detached privileged branch from the updated checkpoint each round. Self-Refined Concise Learning (SRCL) complements this signal by training on shorter, verified refinements of the same on-policy response. Their joint update produces the checkpoint used as both the student and privileged teacher in the next round, closing the recursive loop illustrated in fig. 2 .

### 3.1 Dynamic Co-Evolution