Latent On-Policy Self-Distillation — part 4/4

Cleaned-text code points [46172, 56618). Bibliography omitted here; source.md includes it.

#### Equivalence verification.

To confirm that the placeholder-based injection introduces no numerical artifacts, we replaced the latent span with known token embeddings from the vocabulary and compared the token-ID input path against the embedding-tensor input path. The two paths produce identical greedy-decoded sequences with negligible log-probability differences, verifying that latent injection is numerically equivalent to standard token-ID forwarding.

#### Encoder LoRA isolation.

The encoder Enc ψ \operatorname{Enc}_{\psi} uses a LoRA adapter to specialize hidden-state extraction for experience compression. This adapter is explicitly disabled during the main forward pass (both at inference and during teacher evaluation in training), so that the model behaves as a pure base actor conditioned on the injected latent context. This dual-use pattern—LoRA active for encoding, inactive for generation—avoids interference between the two roles.

### A.5 Prompt and Interaction Format

#### Agentic system prompt.

For tool-use tasks (EnvScaler, ACEBench), the agent receives a system prompt instructing it to complete tasks via step-by-step tool invocation:

#### Coding system prompt.

For TACO and LiveCodeBench, the model receives a minimal system instruction:

HumanEval+ and MBPP+ use no system prompt, following the official EvalPlus evaluation protocol. All coding benchmarks use Python as the target language.

#### Latent-context framing text.

The latent privileged context is injected into the first user message, wrapped by domain-specific framing text. The framing instructs the model to treat the injected content as a reference from a different task, not as instructions to follow verbatim.

The latent tokens produced by the composer replace the framing’s interior (as described in Section A.4 ).

#### Encoder input format.

When the encoder Enc ψ \operatorname{Enc}_{\psi} processes a retrieved experience for compression, the input is formatted as “ Task to solve: \n x x \n\nReference past trajectory: \n m j m_{j} ”, where x x is the current task description and m j m_{j} is the retrieved trajectory. This task-conditional formatting allows the QFormer to attend to task-relevant features when producing the latent tokens.

## Appendix B Experiment Details

### B.1 Baseline Setup

All trainable methods share the same base model, training task distribution, on-policy rollout budget ( 32 32 rollouts per step), optimizer (AdamW, learning rate 10 − 5 10^{-5} , gradient clipping 1.0 1.0 ), and evaluation protocol. Each distillation-based method uses the KL direction prescribed by its original paper: OPSD and SDFT use forward KL over the full vocabulary, SDPO uses reverse KL with top- K K truncation and a tail bucket, and Skill-SD uses sampled-token reverse KL with importance weighting. SDFT and OPSD require ground-truth demonstrations as privileged context for the teacher. These are drawn from a shared pool of verified-successful trajectories: for agentic tasks, trajectories must exceed a reward threshold; for coding tasks, solutions must pass all test cases. The pool is first populated from the base model’s own rollouts; when its coverage is insufficient, we synthesize additional ground-truth trajectories by querying stronger models (DeepSeek-V4-Pro and Qwen3.7-Max in our experiments). Below we describe the method-specific configurations.

#### Vanilla.

The unadapted backbone model, evaluated without any post-training.

#### GRPO.

Standard group relative policy optimization ( Shao et al., 2024 ) . Each step samples G = 4 G{=}4 rollouts per task with stochastic decoding; group-normalized advantages weight a PPO-clip objective with ϵ lo = ϵ hi = 0.2 \epsilon_{\mathrm{lo}}{=}\epsilon_{\mathrm{hi}}{=}0.2 . No teacher or privileged context is used; the training signal comes entirely from environment rewards.

#### SDFT.

Demonstration-conditioned distillation ( Shenfeld et al., 2026 ) . The teacher receives oracle trajectories from the shared pool as textual in-context demonstrations appended to the first user message. The teacher is an EMA shadow of the student ( α EMA = 0.01 \alpha_{\mathrm{EMA}}{=}0.01 , synced every step). The distillation loss is forward KL over the full vocabulary, following the official implementation.

#### OPSD.

On-policy self-distillation ( Zhao et al., 2026 ) . The teacher receives oracle trajectories from the shared pool directly as privileged context in the system message. The teacher is frozen throughout training. The distillation loss is forward KL over the full vocabulary, as prescribed by the original paper.

#### SDPO.

Self-distillation policy optimization ( Hübotter et al., 2026 ) . The teacher is an EMA shadow of the student (update rate 0.05 0.05 ), conditioned on successful sibling trajectories sampled within the same training step ( G = 4 G{=}4 rollouts per task). Unlike SDFT and OPSD, SDPO does not use an external trajectory pool; it filters from its own rollouts using the same domain-specific success criterion (reward threshold for agentic tasks, full test-case pass for coding tasks).

#### Skill-SD.

Skill-conditioned self-distillation ( Wang et al., 2026 ) . Each task selects one skill from a skill bank via UCB1 ( c = 2 c{=}\sqrt{2} ). Skills are distilled from rollout trajectories into structured summaries and injected into the teacher’s first user message:

The loss combines PPO-clip ( ϵ lo = 0.2 \epsilon_{\mathrm{lo}}{=}0.2 , ϵ hi = 0.28 \epsilon_{\mathrm{hi}}{=}0.28 ) with importance-weighted sampled-token reverse KL ( λ sdl = 0.001 \lambda_{\mathrm{sdl}}{=}0.001 ), following the original paper.

### B.2 Training Data and Domain Setup

All methods are trained exclusively on the two datasets listed in Table 4 ; no additional task dataset is used. In particular, the LOPD experience bank contains only rollouts from the corresponding training split and excludes every evaluation task and trajectory. The agentic training corpus (EnvScaler) covers diverse tool-interactive scenarios including insurance, logistics, e-commerce, and healthcare, with each task requiring multi-turn API interactions to complete. The coding training corpus (TACO subset of DeepCoder) consists of competitive programming problems with verified test cases, spanning algorithmic topics such as dynamic programming, graph traversal, and string manipulation. The environment reward differs by domain: EnvScaler returns a continuous reward in [ 0 , 1 ] [0,1] reflecting the fraction of subtasks completed, while TACO returns a binary reward ( 1 1 if all test cases pass, 0 0 otherwise).

### B.3 Evaluation Protocols

Table 5 summarizes the inference configuration for each benchmark. All benchmarks use thinking mode enabled.

#### EnvScaler.

We evaluate on a held-out set of 200 200 tasks, randomly sampled from the full task pool and fully disjoint from the training split.

#### BFCL-v3.

We use the official Berkeley Function Calling Leaderboard V3 codebase with four multi-turn subsets: base, missing-function, missing-parameter, and long-context. Models are served in function-calling (FC) mode. We report per-subset and average scores.

#### ACEBench.

ACEBench evaluates 50 50 tasks split into multi-step ( 20 20 ) and multi-turn ( 30 30 ) categories. A user simulator ( deepseek-v4-flash via API) drives the multi-turn dialogues.

#### LiveCodeBench.

We use the official code_generation_lite evaluation harness and report pass@1 on release v5 ( 279 279 tasks, Aug 2024–Feb 2025) and release v6 ( 131 131 tasks, Feb–May 2025).

#### EvalPlus.

We use HumanEval+ v0.1.10 ( 164 164 tasks) and MBPP+ v0.2.0 ( 378 378 tasks) from the official EvalPlus codebase.

### B.4 Ablation and Analysis Setup

This section details the configuration of each analysis experiment in Section 4.3 .

#### Effect of Joint Optimization ( Figure 3 ).

Each row trains LOPD with a different margin m m and evaluates the resulting student on the EnvScaler test set without retrieval, the composer, or latent privileged context at inference, isolating the effect of joint optimization on the resulting policy. The “Frozen ϕ 0 \phi_{0} ” row trains with a frozen composer (no joint optimization). All jointly-optimized rows share the same training configuration except for m m .

#### Training Dynamics ( Figure 4 ).

Mean reward is periodically evaluated on the EnvScaler test set throughout training using Qwen3-4B .

#### Sensitivity Analysis ( Figure 5 ).

(a) Latent-token capacity : we train separate LOPD variants with K ∈ { 8 , 16 , 32 , 64 , 128 } K\in\{8,16,32,64,128\} and evaluate each resulting student alone on the EnvScaler test set ( n ret = 3 n_{\mathrm{ret}}{=}3 during training). (b–e) Retrieval count : we train separate LOPD variants with K = 32 K{=}32 , m = 0.05 m{=}0.05 , and n ret ∈ { 1 , … , 10 } n_{\mathrm{ret}}\in\{1,\ldots,10\} , then evaluate the resulting students on the EnvScaler test set and ACEBench without retrieval, the composer, or latent privileged context.

#### Behavioral Internalization ( Table 3 ).

Vanilla : the unadapted Qwen3-4B backbone. Base + Composer : the jointly optimized composer ( m = 0.05 m{=}0.05 ) paired with the unadapted backbone and latent privileged context ( n ret = 3 n_{\mathrm{ret}}{=}3 , K = 32 K{=}32 ). LOPD : the distilled student policy evaluated without retrieval, the composer, or latent privileged context. All three are evaluated on the EnvScaler test set with otherwise identical inference settings.

## Instructions for reporting errors

We are continuing to improve HTML versions of papers, and your feedback helps enhance accessibility and mobile support. To report errors in the HTML that will help us improve conversion and rendering, choose any of the methods listed below:

Click the "Report Issue" ( ) button, located in the page header.

Tip: You can select the relevant text first, to include it in your report.

Our team has already identified the following issues . We appreciate your time reviewing and reporting rendering errors we may not have found yet. Your efforts will help us improve the HTML versions for all readers, because disability should not be a barrier to accessing research. Thank you for your continued support in championing open access for all.

Have a free development cycle? Help support accessibility at arXiv! Our collaborators at LaTeXML maintain a list of packages that need conversion , and welcome developer contributions .