Latent On-Policy Self-Distillation — part 3/4

Cleaned-text code points [30189, 46172). Bibliography omitted here; source.md includes it.

### 4.3 Framework Analysis

#### Effect of Joint Optimization.

Does joint optimization actually produce a better privileged context, or merely introduce additional trainable parameters? This is the central ablation behind our claim. Figure 3 compares a frozen composer with jointly optimized variants and evaluates each resulting student without retrieved experience or latent context, so the difference reflects what context learning transfers into the policy. Training with a frozen composer ( ϕ 0 \phi_{0} ) achieves 0.573 0.573 . Without the margin constraint ( m = 0 m{=}0 ), the student drops to 0.551 0.551 , indicating that unconstrained distillation gradients degrade the latent context. Weak margins ( m ≤ 0.01 m{\leq}0.01 ) do not prevent this decline. With m ≥ 0.02 m{\geq}0.02 , the student surpasses the frozen-composer baseline, reaching 0.637 0.637 at m = 0.05 m{=}0.05 and 0.626 0.626 at m = 0.10 m{=}0.10 . The improvement suggests that the distillation process exposes the composer to a signal unavailable during initialization: what evidence the current student’s trajectory distribution requires from the privileged context. The margin constraint is necessary to realize this benefit—without it, collapsing the teacher toward the student is a lower-resistance path than learning a more informative representation.

#### Training Dynamics and Sample Efficiency.

Beyond final performance, a practical question is whether LOPD reaches strong performance with fewer on-policy generations. To test this, Figure 4 tracks the strongest baselines over the same 1,600 1{,}600 -generation budget. LOPD exceeds 0.61 0.61 mean reward after 320 320 generations and reaches 0.637 0.637 by generation 576 576 . Its reward then remains in a narrow 0.63 0.63 – 0.64 0.64 range through generation 1,600 1{,}600 , showing that the early gain is sustained rather than caused by a shorter training horizon. GRPO and Skill-SD improve more gradually and finish at 0.611 0.611 and 0.588 0.588 , respectively. The persistent early separation suggests that the latent teacher extracts a denser learning signal from each visited trajectory.

#### Sensitivity Analysis.

How much latent capacity and retrieved experience does the learnable substrate actually need? We vary the number of latent tokens produced by the composer and the number of experiences retrieved during training, then evaluate the resulting student alone. Figure 5 (a) shows a capacity threshold in the latent bottleneck. EnvScaler reward remains near 0.56 0.56 with 8 8 or 16 16 tokens per experience, rises sharply to 0.637 0.637 with 32 32 , and then fluctuates without a consistent gain at 64 64 and 128 128 . We therefore use K = 32 K{=}32 tokens per experience as the smallest setting that escapes the low-capacity regime. Retrieval sensitivity is shown in Figure 5 (b–e), where n ret n_{\mathrm{ret}} varies from 1 1 to 10 10 . EnvScaler reward improves from 0.605 0.605 with one retrieval to 0.637 0.637 with three, but additional retrievals yield no monotonic benefit. At the default n ret = 3 n_{\mathrm{ret}}{=}3 , ACEBench reaches 56.6 56.6 on M-Step, 63.3 63.3 on M-Turn, and 60.6 60.6 overall, matching the main result in Table 1 . Larger retrieval counts can improve one ACEBench regime without producing the same trajectory in the other, and the aggregate remains on a broad plateau rather than varying monotonically. We therefore retain n ret = 3 n_{\mathrm{ret}}{=}3 as the earliest setting that attains the strongest EnvScaler reward while already reaching competitive ACEBench performance.

#### Behavioral Internalization.

Finally, higher reward alone does not reveal what the student has internalized. We therefore ask whether LOPD changes how the student interacts with the environment. Table 3 shows that the distilled student inherits the interaction pattern induced by latent context. Relative to the vanilla model, LOPD uses more environment steps ( 17.04 17.04 vs. 11.12 11.12 ) but makes far fewer tool calls per step ( 1.11 1.11 vs. 3.50 3.50 ), indicating a shift from issuing many speculative calls at once to executing a more sequential plan. Its first-step response is 37.5 % 37.5\% shorter, repeated calls fall from 8.89 8.89 to 5.25 5.25 , and reward per tool call rises from 0.038 0.038 to 0.050 0.050 . The latent-context-conditioned base model and the final student exhibit the same qualitative pattern, providing behavioral evidence that LOPD internalizes the teacher’s procedural guidance rather than merely fitting the aggregate reward. Per-experiment configurations are detailed in Section B.4 .

### 4.4 Case Study

Having established that the latent context improves learning, a natural question is what these continuous tokens actually encode. To obtain a qualitative view, we apply the frozen language-model head to each of the 32 32 latent tokens and inspect ten tool-use and ten coding examples, with and without task conditioning. Figure 6 shows two representative cases. In the agentic example, the current task and retrieved trajectory share the same add–update–change–withdraw operation schema despite using different entities. In the coding example, the retrieved Josephus recurrence matches the circular-elimination structure of the target problem. Nevertheless, both projections remain fragmented mixtures of multilingual and code-like tokens, and task conditioning changes the surface projection without yielding a readable procedure or copying the retrieved solution. This is consistent with a distributed latent representation, although direct decodability alone does not establish which information the teacher functionally uses.

## 5 Conclusion

The question behind this work is not which new artifact should be appended to an OPSD teacher, but whether the teacher’s privileged context can itself be learned from experience. LOPD answers this question by transforming retrieved raw trajectories into differentiable latent privileged context, using the resulting self-teacher to supervise the student’s own visited prefixes, and constraining the learned context to preserve a verifiable teacher advantage. Across agentic tool use and code generation, this formulation achieves the best aggregate result in all ten backbone–benchmark comparisons and surpasses GRPO and Skill-SD with less than 30 % 30\% of their rollout budget. The analyses sharpen this interpretation: retrieval alone is insufficient, unconstrained joint optimization can collapse the teacher toward the student, and the privileged margin turns context learning into productive supervision. The induced behavioral shift remains in the trained student, which acts without privileged context. More broadly, scalable self-evolution should not depend on a succession of increasingly elaborate, human-authored experience formats. Raw trajectories provide a minimal substrate, and richer repositories or retrievers may broaden what is available, but learning should decide what becomes useful guidance. In this sense, LOPD is less another privileged-context recipe than evidence for a different design principle: experience representations should be optimized end-to-end for the policies they are meant to improve.

## Appendix A Implementation Details

### A.1 Composer Architecture

The latent composer Φ ϕ \Phi_{\phi} consists of an encoder Enc ψ \operatorname{Enc}_{\psi} and a compressor Comp χ \operatorname{Comp}_{\chi} , with ϕ ≔ ( ψ , χ ) \phi\coloneqq(\psi,\chi) . The encoder uses the frozen backbone augmented with a trainable LoRA adapter (rank 8 8 , α = 16 \alpha{=}16 , dropout 0 0 , applied to all seven linear projections per transformer block: 𝐖 q \mathbf{W}_{q} , 𝐖 k \mathbf{W}_{k} , 𝐖 v \mathbf{W}_{v} , 𝐖 o \mathbf{W}_{o} , gate, up, and down projections). During encoding, the task description is prepended to each retrieved experience to enable task-conditional compression, so that the QFormer’s cross-attention can condition on the current task when compressing the experience (see Section A.5 for the exact format).

The compressor is a QFormer-style perceiver where learned queries cross-attend to the encoder’s hidden states. It has 8 8 cross-attention layers with shared parameters (i.e., all layers reuse the same weights), attention heads and hidden dimension inherited from the backbone model, and a feed-forward multiplier of 4 4 . Learned queries 𝐐 χ ∈ ℝ K × d \mathbf{Q}_{\chi}\in\mathbb{R}^{K\times d} are initialized from 𝒩 ⁡ ( 0 , 1 / d ) \mathcal{N}(0,1/\sqrt{d}) . The compressor operates independently on each retrieved experience, producing K K latent tokens per item; the outputs are concatenated to form the full latent context 𝒄 ϕ \bm{c}_{\phi} . Based on the sensitivity analysis in Section 4.3 , we set K = 32 K{=}32 tokens per retrieved experience for the final LOPD training runs. The encoder LoRA and QFormer are updated during both cold-start and joint optimization; the backbone weights remain frozen throughout.

### A.2 Cold-Start Initialization

The composer is cold-started by supervised finetuning on retrieved-experience–trajectory pairs with frozen backbone. The training objective is next-token NLL on assistant tokens with latent-augmented input: ℒ init ​ ( ϕ ) = − 𝔼 ( x , 𝒚 ⋆ ) ∼ 𝒟 exp , ℰ = Ret ⁡ ( x ) ​ [ log ⁡ π θ ¯ ​ ( 𝒚 ⋆ ∣ 𝒔 , 𝒄 ϕ ​ ( x , ℰ ) ) ] , \mathcal{L}_{\mathrm{init}}(\phi)=-\mathbb{E}_{(x,\bm{y}^{\star})\sim\mathcal{D}_{\mathrm{exp}},\;\mathcal{E}=\operatorname{Ret}(x)}\!\left[\log\pi_{\bar{\theta}}\!\left(\bm{y}^{\star}\mid\bm{s},\bm{c}_{\phi}(x,\mathcal{E})\right)\right], (14) where 𝒚 ⋆ \bm{y}^{\star} is a successful trajectory and θ ¯ \bar{\theta} denotes the frozen backbone. Concretely, the composer first produces latent tokens 𝒄 ϕ \bm{c}_{\phi} from retrieved experiences and inserts them into the input embedding sequence at designated placeholder positions. The frozen backbone then performs a standard forward pass over this latent-augmented input. Although the backbone parameters receive no gradient updates, the loss gradient flows backward through the frozen layers’ activations to the latent token positions, and from there to the QFormer and LoRA parameters—the same mechanism as prefix-tuning.

The training data is synthesized from the base model’s own rollouts—no external expert or stronger model is required—and filtered by task success. The data volume and number of training steps are adjusted per domain. Training uses AdamW with learning rate 10 − 5 10^{-5} , batch size 8 8 , gradient clipping 3.0 3.0 , task-conditional compression, and n ret = 3 n_{\mathrm{ret}}{=}3 retrieved experiences per task. The LoRA adapter and QFormer are updated during cold-start and remain trainable during subsequent joint optimization; the backbone remains frozen.

### A.3 Retrieval Configuration

Let ℬ = { ( x i , 𝝉 i ) } i = 1 | ℬ | \mathcal{B}=\{(x_{i},\bm{\tau}_{i})\}_{i=1}^{|\mathcal{B}|} denote the experience bank, where each entry pairs a task description x i x_{i} with a successful trajectory 𝝉 i \bm{\tau}_{i} . A dense encoder f ret f_{\mathrm{ret}} maps text to a unit-norm embedding. We adopt asymmetric encoding: document-side embeddings encode the concatenation of task description and trajectory, while query-side embeddings encode only the task description with an instruction-aware query prompt: 𝒛 i doc = f ret ​ ( [ x i ; 𝝉 i ] ) ‖ f ret ​ ( [ x i ; 𝝉 i ] ) ‖ 2 , 𝒛 q = f ret query ​ ( x ) ‖ f ret query ​ ( x ) ‖ 2 , 𝒛 i doc , 𝒛 q ∈ ℝ d ret . \bm{z}_{i}^{\mathrm{doc}}=\frac{f_{\mathrm{ret}}([x_{i};\bm{\tau}_{i}])}{\|f_{\mathrm{ret}}([x_{i};\bm{\tau}_{i}])\|_{2}},\qquad\bm{z}_{q}=\frac{f_{\mathrm{ret}}^{\mathrm{query}}(x)}{\|f_{\mathrm{ret}}^{\mathrm{query}}(x)\|_{2}},\qquad\bm{z}_{i}^{\mathrm{doc}},\,\bm{z}_{q}\in\mathbb{R}^{d_{\mathrm{ret}}}. (15) Given a query task x x , retrieval returns the n ret n_{\mathrm{ret}} entries with the highest cosine similarity: Ret ⁡ ( x ) = top ​ - ​ n ret ⁡ { ( x i , 𝝉 i ) ∈ ℬ | 𝒛 q ⊤ ​ 𝒛 i doc } . \operatorname{Ret}(x)=\operatorname{top\text{-}n_{\mathrm{ret}}}\left\{(x_{i},\bm{\tau}_{i})\in\mathcal{B}\;\middle|\;{\bm{z}_{q}}^{\top}\bm{z}_{i}^{\mathrm{doc}}\right\}. (16) We use Qwen3-Embedding-8B as f ret f_{\mathrm{ret}} , producing d ret = 4,096 d_{\mathrm{ret}}{=}4{,}096 -dimensional embeddings. The index is implemented as a FAISS IndexFlatIP for exact inner-product search over the precomputed document embeddings. The bank ℬ \mathcal{B} is constructed offline exclusively from successful rollouts generated on the training split: agentic trajectories must exceed a task-reward threshold, while coding trajectories must pass all test cases. No evaluation task, trajectory, or outcome is ever inserted into the bank, and the bank is frozen before evaluation. Consequently, LOPD receives neither evaluation-set information nor an additional task corpus; it uses only experience produced from the same training-task split available to the baselines. Trajectories are stored in an observation-lite format that omits verbose environment observations to reduce storage and encoding length. All query embeddings 𝒛 q \bm{z}_{q} are precomputed and cached per task, so the retriever does not need to be loaded during training or evaluation.

Each bank entry stores the task description followed by a linearized action–result trace in observation-lite format, as illustrated below:

### A.4 Latent Injection Mechanism

The latent tokens produced by the composer must be injected into the backbone’s input in a way compatible with the inference engine’s pipeline. Unlike prior latent-state injection work that typically relies on HuggingFace Transformers or TRL for direct inputs_embeds manipulation, our implementation operates on top of production inference engines (SGLang and vLLM).

#### Placeholder strategy.

A text-level sentinel string <|LATENT_PH|> is inserted into the first user message of the chat template, sandwiched between natural-language framing text (e.g., “ The following is a reference example… ” before and “ Now complete your task… ” after). The full prompt is then rendered via the tokenizer’s chat template and split on the sentinel. Each half is tokenized independently, and J ⋅ K J\cdot K dummy token IDs (using the tokenizer’s pad token) are inserted between them: input ​ _ ​ ids = [ t 1 , … , t a ⏟ before , t pad , … , t pad ⏟ J ⋅ K ​ placeholders , t a + 1 , … , t L ⏟ after ] . \mathrm{input\_ids}=[\,\underbrace{t_{1},\ldots,t_{a}}_{\text{before}},\;\underbrace{t_{\mathrm{pad}},\ldots,t_{\mathrm{pad}}}_{J\cdot K\text{ placeholders}},\;\underbrace{t_{a+1},\ldots,t_{L}}_{\text{after}}\,]. (17) This produces a contiguous span of placeholder positions at known absolute indices, without any tokenizer modification.

#### Embedding replacement.

The engine embeds the full input_ids as usual, producing 𝐗 ∈ ℝ L × d \mathbf{X}\in\mathbb{R}^{L\times d} . Before the first transformer layer, the placeholder embeddings are replaced with the latent tokens from the composer: 𝐗 [ p k ] ← ⟨ e k ⟩ , k = 1 , … , J ⋅ K . \mathbf{X}[p_{k}]\leftarrow\langle e_{k}\rangle,\quad k=1,\ldots,J\cdot K. (18) The replacement uses torch.cat over slices (rather than in-place assignment) so that autograd can track gradients through ⟨ e k ⟩ \langle e_{k}\rangle back to the composer parameters during training. The modified embedding tensor is then passed through the backbone’s transformer layers without any architectural change; the operation is transparent to the engine’s KV-cache, attention mask, and batching logic.

#### Engine-specific implementations.

SGLang provides a positional_embed_overrides interface that accepts embeddings at specified absolute positions and performs the replacement in-GPU. vLLM provides a prompt_embeds interface: the full sequence is first embedded on CPU via a frozen copy of the embedding table, the latent positions are overwritten with the composer’s output, and the complete embedding tensor is submitted to the engine. In both cases, the backbone model is unmodified.