OPRD: On-Policy Representation Distillation — part 9/9

Cleaned-text code points [124603, 133588). Bibliography omitted here; source.md includes it.

Let h S ∈ ℝ d S h_{S}\in\mathbb{R}^{d_{S}} denote the student hidden state at the same position. We model the cross-model alignment via the per-direction correlation : after projecting both models into the teacher’s principal coordinate system, the correlation between teacher component i i and the best linear prediction from the student is ρ i ∈ [ 0 , 1 ] \rho_{i}\in[0,1] . Formally, let z T = V ⊤ ​ ( h T − μ T ) ∈ ℝ d T z_{T}=V^{\top}(h_{T}-\mu_{T})\in\mathbb{R}^{d_{T}} be the teacher’s whitened coordinates. Then ρ i 2 ≜ R 2 ​ ( z T , i ∣ h S ) \rho_{i}^{2}\triangleq R^{2}(z_{T,i}\mid h_{S}) , the coefficient of determination of the best linear predictor of the i i -th teacher component from the student.

###### Assumption 2 (Monotone alignment decay) .

The per-direction alignment ρ i \rho_{i} is non-increasing in i i : ρ 1 ≥ ρ 2 ≥ ⋯ ≥ ρ d T \rho_{1}\geq\rho_{2}\geq\cdots\geq\rho_{d_{T}} . That is, the principal (high-variance) directions of the teacher are better predicted by the student than the minor (low-variance) directions.

This assumption is empirically verified in our experiments: the rank-cosine curve ( Figure 12 ) shows that alignment concentrates in the top principal components and degrades monotonically as rank increases.

#### B.6.2 Rate–Distortion Optimality

###### Theorem 9 (PCA bridge is rate–distortion optimal) .

Assume h T ∼ 𝒩 ⁡ ( μ T , Σ T ) h_{T}\sim\mathcal{N}(\mu_{T},\Sigma_{T}) . Among all rank- r r linear encoders P ∈ ℝ r × d T P\in\mathbb{R}^{r\times d_{T}} , the PCA projection P ∗ = [ v 1 , … , v r ] ⊤ P^{*}=[v_{1},\ldots,v_{r}]^{\top} minimizes the reconstruction distortion: P ∗ = arg ⁡ min P ∈ ℝ r × d T rank ⁡ ( P ) = r ⁡ 𝔼 ⁡ [ ‖ h T − μ T − P ⊤ ​ P ​ ( h T − μ T ) ‖ 2 ] . P^{*}=\arg\min_{\begin{subarray}{c}P\in\mathbb{R}^{r\times d_{T}}\\ \mathrm{rank}(P)=r\end{subarray}}\;\mathbb{E}\left[\|h_{T}-\mu_{T}-P^{\top}P(h_{T}-\mu_{T})\|^{2}\right]. The minimum distortion is D ∗ ​ ( r ) = ∑ i = r + 1 d T λ i D^{*}(r)=\sum_{i=r+1}^{d_{T}}\lambda_{i} .

###### Proof.

This is a direct consequence of the Eckart–Young–Mirsky theorem applied to the centered data matrix, or equivalently, of the classical result that PCA minimizes mean-squared reconstruction error among all linear rank- r r projections. For Gaussian sources, this coincides with the rate–distortion function under squared-error distortion at rate R = 1 2 ​ ∑ i = 1 r log ⁡ ( λ i / θ ) R=\frac{1}{2}\sum_{i=1}^{r}\log(\lambda_{i}/\theta) (water-filling), where θ \theta is chosen so that exactly r r components are active. The optimal encoder projects onto the top- r r eigenvectors of Σ T \Sigma_{T} , which is precisely our PCA basis P T P_{T} . ∎

Interpretation. The PCA bridge is not merely a convenient heuristic; it is the information-theoretically optimal linear channel at capacity r r . Any other rank- r r projection would discard more teacher information.

#### B.6.3 Bias–Variance Decomposition of Distillation Error

We now analyze the distillation error (not reconstruction error), which accounts for the student’s ability to match the projected teacher.

###### Definition 4 (Distillation error at rank r r ) .

Given the PCA bridge P T = [ v 1 , … , v r ] ⊤ P_{T}=[v_{1},\ldots,v_{r}]^{\top} and an optimal student projector P S P_{S} (trained to minimize MSE in the r r -dimensional subspace), the expected per-position distillation error is: ℰ ⁡ ( r ) ≜ 𝔼 ⁡ [ ‖ P S ​ h S − P T ​ ( h T − μ T ) ‖ 2 ] . \mathcal{E}(r)\triangleq\mathbb{E}\left[\|P_{S}h_{S}-P_{T}(h_{T}-\mu_{T})\|^{2}\right].

###### Theorem 10 (Bias–variance decomposition) .

Under Assumption 2 and assuming P S P_{S} is the population-optimal linear map from h S h_{S} to the r r -dimensional bridge space, the total distillation error (including the contribution from discarded dimensions) decomposes as: ℰ total ​ ( r ) = ∑ i > r λ i ​ ρ i 2 ⏟ Bias ​ B ​ ( r ) + ∑ i = 1 r λ i ​ ( 1 − ρ i 2 ) ⏟ Variance ​ V ​ ( r ) , \mathcal{E}_{\mathrm{total}}(r)=\underbrace{\sum_{i>r}\lambda_{i}\,\rho_{i}^{2}}_{\text{Bias }B(r)}\;+\;\underbrace{\sum_{i=1}^{r}\lambda_{i}\,(1-\rho_{i}^{2})}_{\text{Variance }V(r)}, (30) where: • B ⁡ ( r ) = ∑ i > r λ i ​ ρ i 2 B(r)=\sum_{i>r}\lambda_{i}\rho_{i}^{2} is the bias : the aligned signal in directions i > r i>r that the bridge discards. This term decreases as r r increases.

• V ⁡ ( r ) = ∑ i = 1 r λ i ​ ( 1 − ρ i 2 ) V(r)=\sum_{i=1}^{r}\lambda_{i}(1-\rho_{i}^{2}) is the variance : the misaligned noise in directions i ≤ r i\leq r that the bridge transmits but the student cannot match. This term increases as r r increases.

Under Assumption 2 , ℰ total ​ ( r ) \mathcal{E}_{\mathrm{total}}(r) is quasi-convex in r r , and the minimizer r ∗ = arg ⁡ min r ​ ℰ total ​ ( r ) r^{*}=\arg\min_{r}\mathcal{E}_{\mathrm{total}}(r) satisfies:

The optimal rank is the direction at which alignment transitions from “mostly signal” to “mostly noise.”

###### Proof.

We decompose the total error into contributions from included and excluded directions.

Step 1: Error from included directions ( i ≤ r i\leq r ). In the r r -dimensional bridge space, the i i -th component of the teacher target is z T , i = v i ⊤ ​ ( h T − μ T ) z_{T,i}=v_{i}^{\top}(h_{T}-\mu_{T}) , which has variance λ i \lambda_{i} . The optimal linear predictor from h S h_{S} achieves R 2 = ρ i 2 R^{2}=\rho_{i}^{2} , so the residual variance (prediction error) in direction i i is: 𝔼 [ ( P S h S ) i − z T , i ) 2 ] = λ i ( 1 − ρ i 2 ) . \mathbb{E}[(P_{S}h_{S})_{i}-z_{T,i})^{2}]=\lambda_{i}(1-\rho_{i}^{2}). Summing over included directions gives V ⁡ ( r ) = ∑ i = 1 r λ i ​ ( 1 − ρ i 2 ) V(r)=\sum_{i=1}^{r}\lambda_{i}(1-\rho_{i}^{2}) .

Step 2: Error from excluded directions ( i > r i>r ). Directions i > r i>r are not transmitted through the bridge, so the student receives no supervision along them. However, these directions contain aligned signal (the student could have matched them if they were included). The “lost opportunity” is the signal that was discardable: λ i ​ ρ i 2 \lambda_{i}\rho_{i}^{2} per direction. Summing gives B ⁡ ( r ) = ∑ i > r λ i ​ ρ i 2 B(r)=\sum_{i>r}\lambda_{i}\rho_{i}^{2} .

Step 3: Optimality condition. The marginal effect of including direction r r is: Δ ⁡ ( r ) = ℰ total ​ ( r ) − ℰ total ​ ( r − 1 ) = λ r ​ ( 1 − ρ r 2 ) − λ r ​ ρ r 2 = λ r ​ ( 1 − 2 ​ ρ r 2 ) . \Delta(r)=\mathcal{E}_{\mathrm{total}}(r)-\mathcal{E}_{\mathrm{total}}(r-1)=\lambda_{r}(1-\rho_{r}^{2})-\lambda_{r}\rho_{r}^{2}=\lambda_{r}(1-2\rho_{r}^{2}). This is negative (including direction r r helps) when ρ r 2 > 1 / 2 \rho_{r}^{2}>1/2 and positive (including direction r r hurts) when ρ r 2 < 1 / 2 \rho_{r}^{2}<1/2 . Under Assumption 2 , ρ i 2 \rho_{i}^{2} is non-increasing, so Δ ⁡ ( r ) \Delta(r) transitions from negative to positive exactly once, establishing quasi-convexity and the optimality condition ρ r ∗ 2 ≈ 1 / 2 \rho_{r^{*}}^{2}\approx 1/2 . ∎

Connection to the low-pass filter analogy. The bias–variance decomposition provides a precise formulation of the low-pass filter intuition from § 3.3.4 : • The “cutoff frequency” of the filter is r ∗ r^{*} : directions below r ∗ r^{*} are “passed” (high ρ i \rho_{i} , mostly signal), directions above r ∗ r^{*} are “stopped” (low ρ i \rho_{i} , mostly noise).

• The “passband ripple” is V ⁡ ( r ∗ ) V(r^{*}) : even within the passed directions, some noise leaks through.

• The “stopband leakage” is B ⁡ ( r ∗ ) B(r^{*}) : even among the stopped directions, some signal is lost.

• The optimal filter (optimal r ∗ r^{*} ) minimizes the sum of ripple and leakage.

Empirical verification. In our Qwen3-4B → \to Qwen3-1.7B experiments, the rank-cosine curve ( Figure 12 ) peaks at r = 8 r\!=\!8 and declines thereafter, consistent with the prediction that r ∗ ≈ 8 r^{*}\approx 8 is the point where ρ i 2 \rho_{i}^{2} crosses 1 / 2 1/2 . The rapid spectral decay of Σ T \Sigma_{T} (verifiable from the PCA explained-variance ratio) ensures that both B ⁡ ( r ∗ ) B(r^{*}) and V ⁡ ( r ∗ ) V(r^{*}) are small at this operating point.

## Instructions for reporting errors

We are continuing to improve HTML versions of papers, and your feedback helps enhance accessibility and mobile support. To report errors in the HTML that will help us improve conversion and rendering, choose any of the methods listed below:

Click the "Report Issue" ( ) button, located in the page header.

Tip: You can select the relevant text first, to include it in your report.

Our team has already identified the following issues . We appreciate your time reviewing and reporting rendering errors we may not have found yet. Your efforts will help us improve the HTML versions for all readers, because disability should not be a barrier to accessing research. Thank you for your continued support in championing open access for all.

Have a free development cycle? Help support accessibility at arXiv! Our collaborators at LaTeXML maintain a list of packages that need conversion , and welcome developer contributions .