Latent On-Policy Self-Distillation — part 1/4

Cleaned-text code points [0, 14206). Bibliography omitted here; source.md includes it.

# Latent On-Policy Self-Distillation

###### Abstract

Enabling agents to learn from experience and internalize it into their policy has become a central problem in self-evolving AI. On-policy self-distillation (OPSD) offers an effective pathway by using a privileged self-teacher to provide dense supervision on the student’s own trajectories; however, existing methods still rely heavily on designer-specified privileged artifacts ( e.g. , answers, feedback, skills, or trajectories), limiting the end-to-end learnability and scalability required for continual self-improvement. In this work, we introduce Latent On-Policy Self-Distillation ( LOPD ), which, rather than proposing another hand-crafted OPSD variant with a newly prescribed form of privileged context, makes the teacher’s privileged context itself learnable end-to-end from experience. Technically, LOPD retrieves relevant experiences and composes them into continuous latent tokens that condition a self-teacher, while the student generates trajectories from the task and interaction history and receives dense token-level supervision at every visited prefix. We further introduce a privileged-margin objective to stabilize and regulate the learning of latent context. Empirically, LOPD demonstrates (I) strong performance , outperforming RLVR and representative OPSD methods including OPSD, SDPO, and Skill-SD across both agentic tool use and code generation; and (II) high learning efficiency , surpassing GRPO and Skill-SD with less than 30 % 30\% of their rollout budget. Ablation studies further provide direct evidence that making privileged context learnable is necessary for realizing these gains. Together, these results position LOPD as a step toward a more scalable and self-directed paradigm for agent evolution.

## 1 Introduction

How should an agent learn from experience? A natural answer is to expose the model to prior successful trajectories, corrective feedback, or expert demonstrations, and then train it to internalize the behaviors they reveal. This intuition has made on-policy distillation (OPD) a central paradigm for experience learning in large language models: the student first samples its own trajectory, and a teacher then provides dense token-level supervision on the states the student actually visits ( Song & Zheng, 2026 ; Li et al., 2026b ) . Compared with off-policy imitation, OPD reduces the mismatch between training and inference; compared with reinforcement learning with verifiable rewards (RLVR), it converts sparse outcome feedback into fine-grained distributional signals over the student’s own rollout ( Hübotter et al., 2026 ; Yang et al., 2026 ) . In this view, OPD is not merely a compression technique for a stronger model, but a mechanism for turning external experience into persistent policy improvement.

On-policy self-distillation (OPSD) has recently emerged as an especially appealing form of this paradigm because it removes the dependence on a separate, stronger teacher. Instead, the teacher and the student are instantiated from the same model under different contexts: the student acts from the task and interaction history, while the teacher additionally receives privileged information such as verified reasoning traces, final answers, environment feedback, or successful prior rollouts ( Zhao et al., 2026 ; Hübotter et al., 2026 ; Penaloza et al., 2026 ) . The privileged context makes the teacher distribution more informative than the student’s, enabling dense feedback without querying an external model. This design shifts the central question of OPSD from which teacher is strong enough? to what privileged context should be given to the self-teacher? The answer matters: the context must expose useful experience, make token-level supervision sharper, and yet remain compatible with a student that will not see that context at inference time.

Existing substrates for privileged experience remain imperfect in a more fundamental sense. Although prior work instantiates privileged context in many forms—verified answers, reasoning traces, environment feedback, contrastive evidence, or action-only trajectories ( Zhao et al., 2026 ; Hübotter et al., 2026 ; Yu et al., 2026c ; Penaloza et al., 2026 ; Yang et al., 2026 ) —these contexts are typically hand-designed or rule-extracted transformations of experience. They decide a priori what the teacher should see: an oracle answer, a textual reflection, a successful rollout, a retrieved document, or another discrete artifact. Such substrates can be useful, but they also constrain self-distillation to the information format chosen by the designer, rather than allowing the teacher to learn which aspects of prior experience are actually useful for supervising the student’s current trajectory. This motivates a substrate-level question for OPSD:

To address this challenge, we propose Latent On-Policy Self-Distillation ( LOPD ), a framework that realizes privileged context as a learnable latent substrate . Its training pipeline otherwise follows standard OPSD: the student first rolls out trajectories from the task and interaction history, and the teacher re-evaluates every visited prefix to provide dense token-level distributions, against which the student is optimized through reverse-KL distillation. The key difference is that the teacher is conditioned not on a pre-defined privileged artifact, but on learnable latent context instantiated by a composer that transforms retrieved experiences into compact continuous tokens. Because this context is differentiable, the same training process that improves the student also teaches the composer which aspects of experience to retain and how to organize them into effective teacher supervision. To ensure that this learning makes the teacher more informative rather than merely easier for the student to match, we introduce a privileged-margin constraint that requires the teacher to maintain a verifiable log-probability advantage over the student. After training, only the student is retained: inference requires no experience database, retrieval module, composer, or latent context. In this way, LOPD turns experience from a hand-crafted input artifact into an end-to-end learnable supervision substrate whose benefits are internalized by the student policy.

Our contributions are summarized as follows: • Context Re-formulation. We reformulate privileged context from pre-defined, hand-designed textual artifacts into an end-to-end learnable latent substrate, allowing the teacher to automatically extract task-relevant supervision signals from prior experience.

• Proposed Solution. We develop LOPD , which composes retrieved experiences into continuous latent context for the self-teacher and jointly optimizes the privileged context and student through on-policy distillation. A privileged-margin constraint keeps the learned context informative and prevents the teacher from collapsing toward the student.

• Experimental Validation. LOPD consistently surpasses RLVR and representative OPSD methods over three model backbones and seven benchmarks, and outperforms GRPO and Skill-SD with less than 30 % 30\% of their rollout budget. Further ablation studies directly establish the necessity of jointly learning the privileged context.

## 2 Related Work

#### On-Policy (Self-)Distillation.

On-policy distillation trains a student on trajectories sampled from its own policy while querying a teacher for dense token-level supervision on the visited states ( Song & Zheng, 2026 ; Li et al., 2026b ) . Recent work has applied this recipe broadly: reasoning and efficient post-training ( Wu et al., 2026a ; Jin et al., 2026 ; Fu et al., 2026 ) , long-context modeling ( Zhang et al., 2026a ) , context and knowledge distillation ( Ye et al., 2026 ; Lazaridis et al., 2026 ) , GUI grounding ( Zhang et al., 2026b ) , and multimodal domains such as video grounding ( Li et al., 2026a ) , speech alignment ( Cao et al., 2026 ) , and visual reasoning ( Yuan et al., 2026 ; Liu et al., 2026 ) . Within this landscape, on-policy self-distillation (OPSD) removes the external teacher by instantiating teacher and student from the same model under different contexts: the student acts under the deployable input, while the teacher receives privileged context ( Zhao et al., 2026 ; Hübotter et al., 2026 ; Penaloza et al., 2026 ; Yang et al., 2026 ; Yu et al., 2026c ) . This line is best understood by the form of privileged context it supplies: verified answers and reasoning traces ( Zhao et al., 2026 ; Yang et al., 2026 ) , rich environment feedback or self-revision signals ( Hübotter et al., 2026 ; He et al., 2026 ; Zhang et al., 2026c ) , contrastive or peer-rollout evidence ( Yu et al., 2026a ; Pan et al., 2026 ) , action-only frontier-agent trajectories ( Penaloza et al., 2026 ) , and retrieved or evidence-guided context ( Ye et al., 2026 ; Lazaridis et al., 2026 ) . These designs demonstrate that privileged context is the key interface through which OPSD converts experience into supervision, but the context itself is usually pre-defined as a textual or discrete artifact; in contrast, LOPD makes the privileged context a learnable latent substrate jointly optimized with distillation.

#### Latent Computation.

Latent computation uses continuous latent tokens/embeddings or hidden states as the carrier of LLM computation rather than natural language ( Yu et al., 2026b ; Zhu et al., 2025 ) . In reasoning , latent tokens can expand the model’s internal compute budget ( Deng et al., 2026 ; Hao et al., 2025 ; Amos et al., 2026 ) , letting the model perform deeper deliberation before producing an answer. In memory , latent states can serve as compact carriers of procedural ( Zhang et al., 2025 ; Yu et al., 2026d ) , factual ( Wang et al., 2024 ; Feng et al., 2026 ) , and experiential ( Hou et al., 2026 ; Wu et al., 2026b ) information, preserving reusable experience without exposing long textual traces in the prompt. In planning , latent computation can represent intermediate plans, subgoals, or world-state summaries that guide downstream actions while remaining flexible and differentiable. Our method is conceptually close to latent memory, but uses it in a different role: the latent state is not an inference-time augmentation, but a learnable privileged context through which an on-policy teacher supervises a student that does not observe this context.

## 3 Method

### 3.1 Problem Setup: Privileged Context in OPSD

We consider a multi-turn agent that receives a task x x and interacts with an environment through observations and actions. We call the acting policy the student . At turn t t , the student observes 𝒔 t = ( x , 𝒐 ≤ t , 𝒂 < t ) \bm{s}_{t}=(x,\bm{o}_{\leq t},\bm{a}_{<t}) , where 𝒐 ≤ t \bm{o}_{\leq t} denotes observations so far and 𝒂 < t \bm{a}_{<t} denotes previous actions, and samples actions to form a trajectory: 𝒂 t ∼ π θ S ( ⋅ ∣ 𝒔 t ) , 𝝉 = ( 𝒔 1 , 𝒂 1 , … , 𝒔 | 𝝉 | , 𝒂 | 𝝉 | ) ∼ π θ S ( ⋅ ∣ x ) , \bm{a}_{t}\sim\pi_{\theta}^{S}(\cdot\mid\bm{s}_{t}),\qquad\bm{\tau}=(\bm{s}_{1},\bm{a}_{1},\ldots,\bm{s}_{|\bm{\tau}|},\bm{a}_{|\bm{\tau}|})\sim\pi_{\theta}^{S}(\cdot\mid x), (1) where 𝒂 t = ( a t , 1 , … , a t , L t ) \bm{a}_{t}=(a_{t,1},\ldots,a_{t,L_{t}}) is the tokenized action at turn t t , L t L_{t} is its length, and | 𝝉 | |\bm{\tau}| is the number of turns. OPSD constructs a teacher by giving the same model additional privileged context. Abstractly, this context is obtained from some experience source ℰ \mathcal{E} by a fixed transformation: 𝒄 fix = Φ fix ( x , ℰ ) , π θ T ( ⋅ ∣ 𝒔 t , 𝒄 fix ) = π θ ( ⋅ ∣ [ 𝒔 t ; 𝒄 fix ] ) , \bm{c}_{\mathrm{fix}}=\Phi_{\mathrm{fix}}(x,\mathcal{E}),\qquad\pi_{\theta}^{T}(\cdot\mid\bm{s}_{t},\bm{c}_{\mathrm{fix}})=\pi_{\theta}(\cdot\mid[\bm{s}_{t};\bm{c}_{\mathrm{fix}}]), (2) where ℰ \mathcal{E} may contain answers, traces, feedback, demonstrations, or retrieved trajectories, and Φ fix \Phi_{\mathrm{fix}} is a designer-specified rule that decides what artifact the teacher sees. Given a student trajectory, OPSD matches teacher and student distributions on the same visited prefixes: ℒ OPSD ( θ ) = 𝔼 𝝉 ∼ π θ S [ ∑ t = 1 | 𝝉 | ∑ n = 1 L t D ( sg [ π θ T ( ⋅ ∣ 𝒔 t , 𝒄 fix , a t , < n ) ] ∥ π θ S ( ⋅ ∣ 𝒔 t , a t , < n ) ) ] , \displaystyle\mathcal{L}_{\mathrm{OPSD}}(\theta)=\mathbb{E}_{\bm{\tau}\sim\pi_{\theta}^{S}}\left[\sum_{t=1}^{|\bm{\tau}|}\sum_{n=1}^{L_{t}}D\!\left(\operatorname{sg}\!\left[\pi_{\theta}^{T}(\cdot\mid\bm{s}_{t},\bm{c}_{\mathrm{fix}},a_{t,<n})\right]\middle\|\pi_{\theta}^{S}(\cdot\mid\bm{s}_{t},a_{t,<n})\right)\right], (3) where a t , < n a_{t,<n} is the action prefix before token n n , D D is a token-level divergence, and sg ⁡ [ ⋅ ] \operatorname{sg}[\cdot] denotes stop-gradient. This setup makes the limitation explicit: the supervision quality is bounded by a pre-defined context constructor. We instead parameterize the constructor: 𝒄 ϕ = Φ ϕ ​ ( x , ℰ ) = ⟨ e 1 ⟩ ⊕ ⟨ e 2 ⟩ ⊕ ⋯ ⊕ ⟨ e K ⟩ , \bm{c}_{\phi}=\Phi_{\phi}(x,\mathcal{E})=\langle e_{1}\rangle\oplus\langle e_{2}\rangle\oplus\cdots\oplus\langle e_{K}\rangle, (4) where Φ ϕ \Phi_{\phi} is a learnable composer, ⟨ e i ⟩ \langle e_{i}\rangle is a continuous latent token, K K is the number of latent tokens per retrieved experience, and ⊕ \oplus denotes context concatenation. The goal of LOPD is to make privileged context itself learnable while preserving the defining asymmetry of OPSD: the teacher additionally observes 𝒄 ϕ \bm{c}_{\phi} , whereas the student continues to condition only on 𝒔 t \bm{s}_{t} .

### 3.2 Learnable Latent Privileged Context

LOPD instantiates Φ ϕ \Phi_{\phi} as a latent-context composer. We deliberately begin with a minimal experience pipeline. Offline, we retain successful rollouts in an experience bank, with each entry storing its task description and a compact action–result trace that omits verbose observations. For a current task x x , a dense retriever embeds the query and each stored task–trajectory pair, then returns the top- J J entries under cosine similarity; these entries form ℰ = { m j } j = 1 J \mathcal{E}=\{m_{j}\}_{j=1}^{J} . Full construction and retrieval details are provided in Section A.3 .