When Does LeJEPA Learn a World Model? — part 3/9

Cleaned-text code points [31228, 46970). Bibliography omitted here; source.md includes it.

Symbol Description World Model z , z ′ ∈ ℝ n z,z^{\prime}\in\mathbb{R}^{n} True latent variables and positive pair (second view) n n Latent dimension (and encoder output dimension) ρ ∈ ( 0 , 1 ) \rho\in(0,1) Autocorrelation of the Ornstein–Uhlenbeck transition η ∼ 𝒩 ⁡ ( 0 , I n ) \eta\sim\mathcal{N}(0,I_{n}) Independent transition noise K > 0 K>0 Diffusion coefficient (Sturm–Liouville setting) Maps g g Unknown nonlinear mixing (generative) function; x = g ⁡ ( z ) x=g(z) f f Learned encoder; y = f ⁡ ( x ) y=f(x) h = f ∘ g h=f\circ g Composed map from latent space to representation space J h ​ ( z ) J_{h}(z) Jacobian matrix of h h at z z Linear algebra O ⁡ ( n ) O(n) Orthogonal group (matrices with Q ​ Q ⊤ = I n QQ^{\top}=I_{n} ) Q ∈ O ⁡ ( n ) Q\in O(n) Orthogonal recovery matrix (exact identifiability) Losses and objectives ℒ ⁡ ( h ) \mathcal{L}(h) Alignment loss: 𝔼 ⁡ [ ‖ h ⁡ ( z ′ ) − h ⁡ ( z ) ‖ 2 ] \mathbb{E}[\|h(z^{\prime})-h(z)\|^{2}] ℒ inv \mathcal{L}_{\mathrm{inv}} Empirical invariance (alignment) loss ℒ SIG \mathcal{L}_{\mathrm{SIG}} SIGReg Gaussianity regularizer λ \lambda Regularization weight balancing ℒ SIG \mathcal{L}_{\mathrm{SIG}} and ℒ inv \mathcal{L}_{\mathrm{inv}} Hermite polynomials and spectral decomposition He k ​ ( x ) \mathrm{He}_{k}(x) Probabilist’s Hermite polynomial of degree k k h ^ k ​ ( x ) \hat{h}_{k}(x) Normalized Hermite polynomial: He k ​ ( x ) / k ! \mathrm{He}_{k}(x)/\sqrt{k!} α ∈ ℕ n \alpha\in\mathbb{N}^{n} Multi-index ( α 1 , … , α n ) (\alpha_{1},\ldots,\alpha_{n}) ; | α | = α 1 + ⋯ + α n |\alpha|=\alpha_{1}+\cdots+\alpha_{n} is the total degree H α ​ ( z ) H_{\alpha}(z) Multivariate Hermite basis function: ∏ j h ^ α j ​ ( z j ) \prod_{j}\hat{h}_{\alpha_{j}}(z_{j}) c α ( i ) c_{\alpha}^{(i)} Hermite coefficient of h i h_{i} at multi-index α \alpha w d ( i ) w_{d}^{(i)} Variance fraction of h i h_{i} at degree d d : ∑ | α | = d ( c α ( i ) ) 2 \sum_{|\alpha|=d}(c_{\alpha}^{(i)})^{2} T T Transition operator: ( T ​ φ ) ​ ( z ) = 𝔼 ⁡ [ φ ⁡ ( z ′ ) ∣ z ] (T\varphi)(z)=\mathbb{E}[\varphi(z^{\prime})\mid z] Approximate identifiability δ ≥ 0 \delta\geq 0 Alignment gap (excess loss above the linear optimum) ε ≥ 0 \varepsilon\geq 0 Covariance deviation: ‖ Cov ⁡ ( h ⁡ ( z ) ) − I n ‖ F \|\mathrm{Cov}(h(z))-I_{n}\|_{F} D D Normalized alignment gap: δ / ( 2 ​ ρ ​ ( 1 − ρ ) ) \delta/(2\rho(1-\rho)) Sturm–Liouville theory (Appendices B, E) 𝒟 \mathcal{D} Sturm–Liouville differential operator / process generator

## Appendix A Proof of Theorem 5.1 (Forward Direction)

### A.1 Intuition: Why Should This Force Linearity?

Before the formal proof, let us build intuition for why linearity is forced. A linear orthogonal map h ⁡ ( z ) = Q ​ z h(z)=Qz clearly preserves the Gaussian and achieves correlation ρ \rho per component, giving ℒ = 2 ​ ( 1 − ρ ) ​ n \mathcal{L}=2(1-\rho)n . Could a nonlinear map do better?

Here is a suggestive information-theoretic argument. The variables z z and z ′ z^{\prime} are jointly Gaussian with correlation ρ \rho , so I ⁡ ( z , z ′ ) = − n 2 ​ ln ⁡ ( 1 − ρ 2 ) I(z;z^{\prime})=-\frac{n}{2}\ln(1-\rho^{2}) . For Gaussians, mutual information depends only on the correlation. If a nonlinear h h achieved higher correlation while keeping Gaussian marginals, and if the joint ( h ⁡ ( z ) , h ⁡ ( z ′ ) ) (h(z),h(z^{\prime})) were also Gaussian, then I ⁡ ( h ⁡ ( z ) , h ⁡ ( z ′ ) ) > I ⁡ ( z , z ′ ) I(h(z);h(z^{\prime}))>I(z;z^{\prime}) , violating the data processing inequality. So nonlinear maps cannot beat linear ones.

This argument has the right conclusion but two gaps: (1) a nonlinear h h can produce a non-Gaussian joint even when both marginals are Gaussian, so the mutual information formula does not apply; (2) even if nonlinear maps cannot beat linear ones, perhaps they can tie . To close both gaps, we need to understand precisely how nonlinearity interacts with the transition between positive pairs. The right tool turns out to be the Hermite polynomials, the natural orthogonal basis for functions of Gaussian variables. Just as Fourier analysis decomposes a signal into frequencies, the Hermite expansion decomposes any function into a linear part, a quadratic part, a cubic part, and so on. The key payoff is that the transition between positive pairs acts on each degree separately, and attenuates higher degrees more. This lets us compute the correlation of any function in closed form, closing both gaps simultaneously.

### A.2 Hermite Polynomial Background

The proof rests on the Hermite polynomials, the natural orthogonal basis for functions of Gaussian random variables. This subsection collects the definitions and properties used throughout the paper.

##### Definitions and orthogonality.

The probabilist’s Hermite polynomials He k ​ ( x ) \mathrm{He}_{k}(x) are defined by the Rodrigues formula: He k ( x ) = ( − 1 ) k e x 2 / 2 d k d ​ x k e − x 2 / 2 , k = 0 , 1 , 2 , … \mathrm{He}_{k}(x)=(-1)^{k}\,e^{x^{2}/2}\,\frac{d^{k}}{dx^{k}}\,e^{-x^{2}/2},\qquad k=0,1,2,\ldots (7) The first several are: He 0 ​ ( x ) = 1 , He 1 ​ ( x ) = x , He 2 ​ ( x ) = x 2 − 1 , He 3 ​ ( x ) = x 3 − 3 ​ x \mathrm{He}_{0}(x)=1,\quad\mathrm{He}_{1}(x)=x,\quad\mathrm{He}_{2}(x)=x^{2}-1,\quad\mathrm{He}_{3}(x)=x^{3}-3x (8) Each He k \mathrm{He}_{k} is a degree- k k polynomial. Note that He 2 ​ ( x ) = x 2 − 1 \mathrm{He}_{2}(x)=x^{2}-1 includes a correction term that makes it orthogonal to the constant and linear terms under the Gaussian measure. More generally, these polynomials satisfy the orthogonality relation under the standard Gaussian measure γ \gamma with density p ( x ) = ( 2 π ) − 1 / 2 e − x 2 / 2 p(x)=(2\pi)^{-1/2}e^{-x^{2}/2} : 𝔼 x ∼ γ ​ [ He j ​ ( x ) ​ He k ​ ( x ) ] = k ! ​ δ j ​ k . \mathbb{E}_{x\sim\gamma}\!\left[\mathrm{He}_{j}(x)\,\mathrm{He}_{k}(x)\right]=k!\,\delta_{jk}. (9) Normalizing by h ^ k ​ ( x ) := He k ​ ( x ) / k ! \hat{h}_{k}(x):=\mathrm{He}_{k}(x)/\sqrt{k!} gives an orthonormal basis of L 2 ​ ( ℝ , γ ) L^{2}(\mathbb{R},\gamma) : 𝔼 ⁡ [ h ^ j ​ ( x ) ​ h ^ k ​ ( x ) ] = δ j ​ k \mathbb{E}[\hat{h}_{j}(x)\,\hat{h}_{k}(x)]=\delta_{jk} . The completeness of this basis means that any f ∈ L 2 ​ ( ℝ , γ ) f\in L^{2}(\mathbb{R},\gamma) can be uniquely expanded as f ⁡ ( x ) = ∑ k = 0 ∞ a k ​ h ^ k ​ ( x ) f(x)=\sum_{k=0}^{\infty}a_{k}\hat{h}_{k}(x) with ∑ k a k 2 = 𝔼 ⁡ [ f ​ ( x ) 2 ] < ∞ \sum_{k}a_{k}^{2}=\mathbb{E}[f(x)^{2}]<\infty (Parseval’s identity).

##### Multivariate extension.

For z = ( z 1 , … , z n ) ∈ ℝ n z=(z_{1},\ldots,z_{n})\in\mathbb{R}^{n} with independent standard Gaussian components, the multivariate Hermite basis is constructed as tensor products. For a multi-index α = ( α 1 , … , α n ) ∈ ℕ n \alpha=(\alpha_{1},\ldots,\alpha_{n})\in\mathbb{N}^{n} with total degree | α | = α 1 + ⋯ + α n |\alpha|=\alpha_{1}+\cdots+\alpha_{n} , define: H α ​ ( z ) = ∏ j = 1 n h ^ α j ​ ( z j ) . H_{\alpha}(z)=\prod_{j=1}^{n}\hat{h}_{\alpha_{j}}(z_{j}). (10) The family { H α } α ∈ ℕ n \{H_{\alpha}\}_{\alpha\in\mathbb{N}^{n}} is an orthonormal basis of L 2 ​ ( ℝ n , γ n ) L^{2}(\mathbb{R}^{n},\gamma_{n}) , where γ n \gamma_{n} is the standard Gaussian measure on ℝ n \mathbb{R}^{n} . The degree | α | |\alpha| captures the “order of nonlinearity”: • Degree 0: H 0 ​ ( z ) = 1 H_{0}(z)=1 . The constant function.

• Degree 1: H e j ​ ( z ) = z j H_{e_{j}}(z)=z_{j} for j = 1 , … , n j=1,\ldots,n . The n n linear coordinate functions.

• Degree 2: terms like z i ​ z j z_{i}z_{j} (for i ≠ j i\neq j ) and ( z j 2 − 1 ) / 2 (z_{j}^{2}-1)/\sqrt{2} . Quadratic nonlinearities.

• Degree d d : captures d d -th order nonlinear structure.

##### Hermite expansion of measure-preserving maps.

Any component h i : ℝ n → ℝ h_{i}:\mathbb{R}^{n}\to\mathbb{R} of a map satisfying h ⁡ ( z ) ∼ 𝒩 ⁡ ( 0 , I n ) h(z)\sim\mathcal{N}(0,I_{n}) has 𝔼 ​ [ h i ​ ( z ) ] = 0 \mathbb{E}[h_{i}(z)]=0 and 𝔼 ⁡ [ h i ​ ( z ) 2 ] = 1 \mathbb{E}[h_{i}(z)^{2}]=1 . By completeness, it expands as: h i ​ ( z ) = ∑ | α | ≥ 1 c α ( i ) ​ H α ​ ( z ) , ∑ | α | ≥ 1 ( c α ( i ) ) 2 = 1 . h_{i}(z)=\sum_{|\alpha|\geq 1}c_{\alpha}^{(i)}\,H_{\alpha}(z),\qquad\sum_{|\alpha|\geq 1}(c_{\alpha}^{(i)})^{2}=1. (11) The constant term ( | α | = 0 |\alpha|=0 ) vanishes because 𝔼 ⁡ [ h i ] = 0 \mathbb{E}[h_{i}]=0 . The normalization ∑ ( c α ( i ) ) 2 = 1 \sum(c_{\alpha}^{(i)})^{2}=1 follows from Parseval’s identity and 𝔼 ⁡ [ h i 2 ] = 1 \mathbb{E}[h_{i}^{2}]=1 . Grouping by total degree defines the variance at degree d d : w d ( i ) = ∑ | α | = d ( c α ( i ) ) 2 , ∑ d ≥ 1 w d ( i ) = 1 . w_{d}^{(i)}=\sum_{|\alpha|=d}(c_{\alpha}^{(i)})^{2},\qquad\sum_{d\geq 1}w_{d}^{(i)}=1. (12) If h i h_{i} is a linear function of z z , then w 1 ( i ) = 1 w_{1}^{(i)}=1 and w d ( i ) = 0 w_{d}^{(i)}=0 for all d ≥ 2 d\geq 2 . If h i h_{i} has any nonlinear component, then w d ( i ) > 0 w_{d}^{(i)}>0 for some d ≥ 2 d\geq 2 .

### A.3 Proof

##### Setup and notation.

Since z ∼ 𝒩 ⁡ ( 0 , I n ) z\sim\mathcal{N}(0,I_{n}) , the multivariate Hermite polynomials { H α } α ∈ ℕ n \{H_{\alpha}\}_{\alpha\in\mathbb{N}^{n}} form an orthonormal basis of L 2 ​ ( ℝ n , γ n ) L^{2}(\mathbb{R}^{n},\gamma_{n}) (Appendix A.2 ). Each component h i h_{i} has 𝔼 ​ [ h i ​ ( z ) ] = 0 \mathbb{E}[h_{i}(z)]=0 and 𝔼 ⁡ [ h i ​ ( z ) 2 ] = 1 \mathbb{E}[h_{i}(z)^{2}]=1 by the Gaussianity constraint, and admits the expansion h i ​ ( z ) = ∑ | α | ≥ 1 c α ( i ) ​ H α ​ ( z ) , ∑ | α | ≥ 1 ( c α ( i ) ) 2 = 1 . h_{i}(z)=\sum_{|\alpha|\geq 1}c_{\alpha}^{(i)}\,H_{\alpha}(z),\qquad\sum_{|\alpha|\geq 1}(c_{\alpha}^{(i)})^{2}=1. (13) Grouping by total degree d = | α | d=|\alpha| , let w d ( i ) = ∑ | α | = d ( c α ( i ) ) 2 w_{d}^{(i)}=\sum_{|\alpha|=d}(c_{\alpha}^{(i)})^{2} be the fraction of variance at degree d d . Then w 1 ( i ) + w 2 ( i ) + w 3 ( i ) + ⋯ = 1 w_{1}^{(i)}+w_{2}^{(i)}+w_{3}^{(i)}+\cdots=1 . If h i h_{i} is linear, then w 1 ( i ) = 1 w_{1}^{(i)}=1 ; any nonlinearity spills variance into d ≥ 2 d\geq 2 . By equation ( 3 ), minimizing ℒ \mathcal{L} is equivalent to maximizing ∑ i 𝔼 ⁡ [ h i ​ ( z ′ ) ​ h i ​ ( z ) ] \sum_{i}\mathbb{E}[h_{i}(z^{\prime})h_{i}(z)] . The proof establishes that each term satisfies 𝔼 ⁡ [ h i ​ ( z ′ ) ​ h i ​ ( z ) ] ≤ ρ \mathbb{E}[h_{i}(z^{\prime})h_{i}(z)]\leq\rho , with equality if and only if h i h_{i} is linear. The argument has three main steps. First, we need to know what the transition z → z ′ z\to z^{\prime} does to each Hermite component individually. Second, we need the cross-correlation structure between all components across the two views: does the degree-2 part of h i ​ ( z ′ ) h_{i}(z^{\prime}) correlate with the degree-1 part of h i ​ ( z ) h_{i}(z) ? Third, we combine these to bound the total correlation any function can achieve.

##### Step 1: The transition attenuates each Hermite degree by ρ d \rho^{d} .

We show that 𝔼 η ​ [ He k ​ ( z ′ ) ] = ρ k ​ He k ​ ( z ) \mathbb{E}_{\eta}[\mathrm{He}_{k}(z^{\prime})]=\rho^{k}\,\mathrm{He}_{k}(z) for z ′ = ρ ​ z + 1 − ρ 2 ​ η z^{\prime}=\rho z+\sqrt{1-\rho^{2}}\,\eta with η ∼ 𝒩 ⁡ ( 0 , 1 ) \eta\sim\mathcal{N}(0,1) independent of z ∼ 𝒩 ⁡ ( 0 , 1 ) z\sim\mathcal{N}(0,1) .

We use the exponential generating function for the Hermite polynomials: ∑ k = 0 ∞ t k k ! ​ He k ​ ( x ) = exp ⁡ ( t ​ x − t 2 2 ) . \sum_{k=0}^{\infty}\frac{t^{k}}{k!}\,\mathrm{He}_{k}(x)=\exp\!\left(tx-\frac{t^{2}}{2}\right). (14) This identity encodes all Hermite polynomials into a single exponential, allowing us to compute the effect of noise on all degrees simultaneously.

Evaluate ( 14 ) at x = z ′ = ρ ​ z + 1 − ρ 2 ​ η x=z^{\prime}=\rho z+\sqrt{1-\rho^{2}}\,\eta : ∑ k = 0 ∞ t k k ! ​ He k ​ ( z ′ ) = exp ⁡ ( t ⁡ ( ρ ​ z + 1 − ρ 2 ​ η ) − t 2 2 ) . \sum_{k=0}^{\infty}\frac{t^{k}}{k!}\,\mathrm{He}_{k}(z^{\prime})=\exp\!\left(t(\rho z+\sqrt{1-\rho^{2}}\,\eta)-\frac{t^{2}}{2}\right). (15) Since η \eta is independent of z z , the right side factors: exp ⁡ ( t ​ ρ ​ z − t 2 2 ) ⋅ exp ⁡ ( t ​ 1 − ρ 2 ​ η ) . \exp\!\left(t\rho z-\frac{t^{2}}{2}\right)\cdot\exp\!\left(t\sqrt{1-\rho^{2}}\,\eta\right). (16) The first factor depends only on z z and comes out of 𝔼 η \mathbb{E}_{\eta} . For the second factor, we use the Gaussian moment generating function: if η ∼ 𝒩 ⁡ ( 0 , 1 ) \eta\sim\mathcal{N}(0,1) , then 𝔼 ⁡ [ e s ​ η ] = e s 2 / 2 \mathbb{E}[e^{s\eta}]=e^{s^{2}/2} for any s s . Setting s = t ​ 1 − ρ 2 s=t\sqrt{1-\rho^{2}} : 𝔼 η ​ [ exp ⁡ ( t ​ 1 − ρ 2 ​ η ) ] = exp ⁡ ( t 2 ​ ( 1 − ρ 2 ) 2 ) . \mathbb{E}_{\eta}\!\left[\exp\!\left(t\sqrt{1-\rho^{2}}\,\eta\right)\right]=\exp\!\left(\frac{t^{2}(1-\rho^{2})}{2}\right). (17) Multiplying the two pieces, the exponents combine: t ​ ρ ​ z − t 2 2 + t 2 ​ ( 1 − ρ 2 ) 2 = t ​ ρ ​ z − t 2 2 + t 2 2 − t 2 ​ ρ 2 2 = t ​ ρ ​ z − ( t ​ ρ ) 2 2 . t\rho z-\frac{t^{2}}{2}+\frac{t^{2}(1-\rho^{2})}{2}=t\rho z-\frac{t^{2}}{2}+\frac{t^{2}}{2}-\frac{t^{2}\rho^{2}}{2}=t\rho z-\frac{(t\rho)^{2}}{2}. (18) So we obtain: ∑ k = 0 ∞ t k k ! ​ 𝔼 η ​ [ He k ​ ( z ′ ) ] = exp ⁡ ( ( ρ ​ t ) ​ z − ( ρ ​ t ) 2 2 ) . \sum_{k=0}^{\infty}\frac{t^{k}}{k!}\,\mathbb{E}_{\eta}\!\left[\mathrm{He}_{k}(z^{\prime})\right]=\exp\!\left((\rho t)z-\frac{(\rho t)^{2}}{2}\right). (19) The right side of ( 19 ) has exactly the form of the generating function ( 14 ) with t t replaced by ρ ​ t \rho t : exp ⁡ ( ( ρ ​ t ) ​ z − ( ρ ​ t ) 2 2 ) = ∑ k = 0 ∞ ( ρ ​ t ) k k ! ​ He k ​ ( z ) = ∑ k = 0 ∞ t k k ! ​ ρ k ​ He k ​ ( z ) . \exp\!\left((\rho t)z-\frac{(\rho t)^{2}}{2}\right)=\sum_{k=0}^{\infty}\frac{(\rho t)^{k}}{k!}\,\mathrm{He}_{k}(z)=\sum_{k=0}^{\infty}\frac{t^{k}}{k!}\,\rho^{k}\,\mathrm{He}_{k}(z). (20) Matching coefficients of t k / k ! t^{k}/k! : 𝔼 η ​ [ He k ​ ( z ′ ) ] = ρ k ​ He k ​ ( z ) . \mathbb{E}_{\eta}\!\left[\mathrm{He}_{k}(z^{\prime})\right]=\rho^{k}\,\mathrm{He}_{k}(z). (21) Since 0 < ρ < 1 0<\rho<1 , we have ρ 2 < ρ \rho^{2}<\rho , ρ 3 < ρ 2 \rho^{3}<\rho^{2} , and so on. The linear component ( k = 1 k=1 ) survives the transition best; every higher-degree component is strictly more attenuated.

This contraction is specific to Gaussians. The proof succeeds because the Gaussian MGF ( e s 2 / 2 e^{s^{2}/2} ) partially cancels the e − t 2 / 2 e^{-t^{2}/2} in the Hermite generating function, leaving e − t 2 ρ 2 / 2 e^{-t^{2}\rho^{2}/2} , which is the generating function again with t → ρ ​ t t\to\rho t . This replacement is where the ρ k \rho^{k} contraction originates. For non-Gaussian distributions, the transition operator still has a spectral decomposition with ordered eigenvalues (see Appendix F ), but the eigenfunctions are no longer Hermite polynomials and the eigenvalues no longer decay geometrically. In particular, the first eigenfunction is monotonic but generally nonlinear, which is why monotonic (not linear) identifiability holds in the general case.

##### Step 2: Different Hermite components do not interact across views.

Step 1 tells us what happens to each Hermite component in isolation. But to compute the correlation 𝔼 ⁡ [ h i ​ ( z ′ ) ​ h i ​ ( z ) ] \mathbb{E}[h_{i}(z^{\prime})h_{i}(z)] for an arbitrary function h i h_{i} , we also need to know whether different components interact across views: does the degree-2 part of h i ​ ( z ′ ) h_{i}(z^{\prime}) correlate with the degree-1 part of h i ​ ( z ) h_{i}(z) ?

The answer is no. We show that 𝔼 ⁡ [ H α ​ ( z ′ ) ​ H β ​ ( z ) ] = ρ | α | ​ δ α ​ β \mathbb{E}[H_{\alpha}(z^{\prime})\,H_{\beta}(z)]=\rho^{|\alpha|}\,\delta_{\alpha\beta} (Mehler’s formula). The δ α ​ β \delta_{\alpha\beta} means that cross-terms vanish: the degree-1 part of h ⁡ ( z ′ ) h(z^{\prime}) only correlates with the degree-1 part of h ⁡ ( z ) h(z) , the degree-2 part only with degree-2, and so on. Each degree- d d component contributes correlation ρ d \rho^{d} . The Hermite basis diagonalizes the transition.