When Does LeJEPA Learn a World Model? — part 6/9

Cleaned-text code points [78324, 93753). Bibliography omitted here; source.md includes it.

By the change of variables formula for densities, p ⁡ ( h ⁡ ( z ) ) ​ | det J h ​ ( z ) | = p ⁡ ( z ) p(h(z))\,|\det J_{h}(z)|=p(z) for all z z . Substituting the Gaussian density p ( z ) = ( 2 π ) − n / 2 exp ( − ∥ z ∥ 2 / 2 ) p(z)=(2\pi)^{-n/2}\exp(-\|z\|^{2}/2) and noting that the normalizing constants cancel: ln ⁡ | det J h ​ ( z ) | = 1 2 ​ ‖ h ⁡ ( z ) ‖ 2 − 1 2 ​ ‖ z ‖ 2 . \ln|\det J_{h}(z)|=\tfrac{1}{2}\|h(z)\|^{2}-\tfrac{1}{2}\|z\|^{2}. (65) Taking expectations and using 𝔼 ⁡ [ ‖ h ⁡ ( z ) ‖ 2 ] = 𝔼 ⁡ [ ‖ z ‖ 2 ] = n \mathbb{E}[\|h(z)\|^{2}]=\mathbb{E}[\|z\|^{2}]=n (both follow from h ⁡ ( z ) ∼ 𝒩 ⁡ ( 0 , I n ) h(z)\sim\mathcal{N}(0,I_{n}) ): 𝔼 ⁡ [ ln ⁡ | det J h ​ ( z ) | ] = 0 . \mathbb{E}\!\left[\ln|\det J_{h}(z)|\right]=0. (66)

##### Step 3: Lower bound via AM-GM and Jensen.

Let σ 1 ​ ( z ) , … , σ n ​ ( z ) \sigma_{1}(z),\ldots,\sigma_{n}(z) be the singular values of J h ​ ( z ) J_{h}(z) . Then ‖ J h ​ ( z ) ‖ F 2 = ∑ i σ i ​ ( z ) 2 \|J_{h}(z)\|_{F}^{2}=\sum_{i}\sigma_{i}(z)^{2} and | det J h ​ ( z ) | = ∏ i σ i ​ ( z ) |\det J_{h}(z)|=\prod_{i}\sigma_{i}(z) . By AM-GM applied to σ 1 2 , … , σ n 2 \sigma_{1}^{2},\ldots,\sigma_{n}^{2} : 1 n ​ ∑ i = 1 n σ i ​ ( z ) 2 ≥ ( ∏ i = 1 n σ i ​ ( z ) 2 ) 1 / n = | det J h ​ ( z ) | 2 / n , \frac{1}{n}\sum_{i=1}^{n}\sigma_{i}(z)^{2}\geq\left(\prod_{i=1}^{n}\sigma_{i}(z)^{2}\right)^{1/n}=|\det J_{h}(z)|^{2/n}, (67) so ‖ J h ​ ( z ) ‖ F 2 ≥ n ​ | det J h ​ ( z ) | 2 / n \|J_{h}(z)\|_{F}^{2}\geq n\,|\det J_{h}(z)|^{2/n} pointwise. Taking expectations: ℒ ⁡ ( h ) ≥ n ​ 𝔼 ​ [ | det J h ​ ( z ) | 2 / n ] . \mathcal{L}(h)\geq n\,\mathbb{E}\!\left[|\det J_{h}(z)|^{2/n}\right]. (68) Writing | det J h ​ ( z ) | 2 / n = exp ⁡ ( 2 n ​ ln ⁡ | det J h ​ ( z ) | ) |\det J_{h}(z)|^{2/n}=\exp\!\left(\tfrac{2}{n}\ln|\det J_{h}(z)|\right) and applying Jensen’s inequality to the strictly convex function x ↦ e ( 2 / n ) ​ x x\mapsto e^{(2/n)x} : 𝔼 ⁡ [ | det J h ​ ( z ) | 2 / n ] ≥ exp ⁡ ( 2 n ​ 𝔼 ​ [ ln ⁡ | det J h ​ ( z ) | ] ) = exp ⁡ ( 0 ) = 1 . \mathbb{E}\!\left[|\det J_{h}(z)|^{2/n}\right]\geq\exp\!\left(\tfrac{2}{n}\,\mathbb{E}\!\left[\ln|\det J_{h}(z)|\right]\right)=\exp(0)=1. (69) Therefore ℒ ⁡ ( h ) ≥ n \mathcal{L}(h)\geq n . Any orthogonal map h ⁡ ( z ) = Q ​ z h(z)=Qz achieves this bound since ‖ Q ‖ F 2 = n \|Q\|_{F}^{2}=n .

##### Step 4: Equality forces orthogonal Jacobian.

At the minimum ℒ ⁡ ( h ) = n \mathcal{L}(h)=n , equality must hold in both Jensen and AM-GM. Strict convexity of the exponential forces ln ⁡ | det J h ​ ( z ) | \ln|\det J_{h}(z)| to be constant a.e., and since its expectation is 0, we get | det J h ​ ( z ) | = 1 |\det J_{h}(z)|=1 a.e. Equality in AM-GM forces all squared singular values to be equal: σ i ​ ( z ) 2 = ⋯ = σ n ​ ( z ) 2 \sigma_{i}(z)^{2}=\cdots=\sigma_{n}(z)^{2} . Combined with ∏ i σ i = 1 \prod_{i}\sigma_{i}=1 , this gives σ i ​ ( z ) = 1 \sigma_{i}(z)=1 for all i i . Thus: J h ​ ( z ) ⊤ ​ J h ​ ( z ) = I n for a.e. ​ z . J_{h}(z)^{\top}J_{h}(z)=I_{n}\qquad\text{for a.e.\ }z. (70) By C 1 C^{1} continuity, this holds everywhere.

##### Step 5: Orthogonal Jacobian implies linearity (Mazur–Ulam).

Since J h ​ ( z ) ⊤ ​ J h ​ ( z ) = I n J_{h}(z)^{\top}J_{h}(z)=I_{n} everywhere, h h preserves the norm of tangent vectors: ‖ J h ​ ( z ) ​ v ‖ = ‖ v ‖ \|J_{h}(z)v\|=\|v\| for all v v . For any z 1 , z 2 ∈ ℝ n z_{1},z_{2}\in\mathbb{R}^{n} , integrating along the straight-line path γ ⁡ ( t ) = z 1 + t ⁡ ( z 2 − z 1 ) \gamma(t)=z_{1}+t(z_{2}-z_{1}) : h ⁡ ( z 2 ) − h ⁡ ( z 1 ) = ∫ 0 1 J h ​ ( γ ⁡ ( t ) ) ​ ( z 2 − z 1 ) ​ 𝑑 t . h(z_{2})-h(z_{1})=\int_{0}^{1}J_{h}(\gamma(t))(z_{2}-z_{1})\,dt. (71) Taking norms and applying the triangle inequality: ‖ h ⁡ ( z 2 ) − h ⁡ ( z 1 ) ‖ ≤ ∫ 0 1 ‖ J h ​ ( γ ⁡ ( t ) ) ​ ( z 2 − z 1 ) ‖ ​ 𝑑 t = ∫ 0 1 ‖ z 2 − z 1 ‖ ​ 𝑑 t = ‖ z 2 − z 1 ‖ . \|h(z_{2})-h(z_{1})\|\leq\int_{0}^{1}\|J_{h}(\gamma(t))(z_{2}-z_{1})\|\,dt=\int_{0}^{1}\|z_{2}-z_{1}\|\,dt=\|z_{2}-z_{1}\|. (72) Since h h is a diffeomorphism, h − 1 h^{-1} exists and is C 1 C^{1} with J h − 1 ​ ( h ⁡ ( z ) ) = J h ​ ( z ) − 1 = J h ​ ( z ) ⊤ J_{h^{-1}}(h(z))=J_{h}(z)^{-1}=J_{h}(z)^{\top} , which is also orthogonal ( J h ​ J h ⊤ = I n J_{h}J_{h}^{\top}=I_{n} for square matrices with J h ⊤ ​ J h = I n J_{h}^{\top}J_{h}=I_{n} ). Applying the same argument to h − 1 h^{-1} yields ‖ z 2 − z 1 ‖ ≤ ‖ h ⁡ ( z 2 ) − h ⁡ ( z 1 ) ‖ \|z_{2}-z_{1}\|\leq\|h(z_{2})-h(z_{1})\| . Together: ‖ h ⁡ ( z 1 ) − h ⁡ ( z 2 ) ‖ = ‖ z 1 − z 2 ‖ for all ​ z 1 , z 2 ∈ ℝ n , \|h(z_{1})-h(z_{2})\|=\|z_{1}-z_{2}\|\qquad\text{for all }z_{1},z_{2}\in\mathbb{R}^{n}, (73) so h h is a surjective isometry of ℝ n \mathbb{R}^{n} . By the Mazur–Ulam theorem [ 89 ] , every surjective isometry of a real normed space is affine: h ⁡ ( z ) = Q ​ z + b h(z)=Qz+b with Q Q orthogonal and b ∈ ℝ n b\in\mathbb{R}^{n} . Measure preservation forces 0 = 𝔼 ⁡ [ h ⁡ ( z ) ] = Q ​ 𝔼 ​ [ z ] + b = b 0=\mathbb{E}[h(z)]=Q\mathbb{E}[z]+b=b , giving h ⁡ ( z ) = Q ​ z h(z)=Qz . ∎

##### Generalization.

No step in the proof above uses the Gaussian assumption on p p ; the noise η \eta need only satisfy 𝔼 ⁡ [ η ​ η ⊤ ] = I n \mathbb{E}[\eta\eta^{\top}]=I_{n} (Step 1), and Steps 2–5 hold for any density preserved by h h . The identity map always achieves ℒ = n \mathcal{L}=n , so the lower bound is tight regardless of p p . The conclusion is that any C 1 C^{1} measure-preserving diffeomorphism minimizing ℒ \mathcal{L} must be an isometry h ⁡ ( z ) = Q ​ z + b h(z)=Qz+b with p ⁡ ( Q ​ z + b ) = p ⁡ ( z ) p(Qz+b)=p(z) .

### E.3 Comparison of the Two Proofs

The Hermite proof (Thm. 5.1 ) and the Dirichlet energy proof (Thm. E.2 ) establish the same conclusion through different mechanisms:

Hermite proof Dirichlet energy proof Regularity Measurable C 1 C^{1} diffeomorphism Noise level Any ρ ∈ ( 0 , 1 ) \rho\in(0,1) Infinitesimal ( ρ → 1 \rho\to 1 ) Key tool Mehler’s formula AM-GM + Jensen + Mazur–Ulam Mechanism Low-pass filter on degrees Jacobian rigidity Connection to Harmonic analysis Differential geometry

The Hermite proof is strictly more general (weaker assumptions, finite noise), but the Dirichlet energy proof provides geometric intuition that complements the spectral perspective: the Jacobian must be an isometry at every point, so the map cannot compress or stretch any direction anywhere.

## Appendix F Prior Work: Connection to Slow Feature Analysis

Our identifiability result has a deep connection to the theory of Slow Feature Analysis (SFA) developed by Wiskott and Sejnowski [70] and extended to nonlinear blind source separation by Sprekeler et al. [72] . Sobal et al. [71] analyzed the tendency of JEPAs to focus on slow features. We spell out this connection in detail, both because it clarifies the scope of our contribution and because the SFA perspective suggests natural generalizations.

##### SFA and its optimization problem.

SFA seeks scalar output functions g j ​ ( x ) g_{j}(x) whose output signals y j ​ ( t ) = g j ​ ( x ⁡ ( t ) ) y_{j}(t)=g_{j}(x(t)) vary as slowly as possible in time, as measured by the Δ \Delta -value Δ ⁡ ( y j ) = ⟨ y ˙ j 2 ⟩ t \Delta(y_{j})=\langle\dot{y}_{j}^{2}\rangle_{t} , subject to the constraints ⟨ y j ⟩ = 0 \langle y_{j}\rangle=0 (zero mean), ⟨ y j 2 ⟩ = 1 \langle y_{j}^{2}\rangle=1 (unit variance), and ⟨ y i ​ y j ⟩ = 0 \langle y_{i}y_{j}\rangle=0 for i < j i<j (decorrelation) [ 70 ] . For discrete-time data, the Δ \Delta -value is proportional to ⟨ y 2 ⟩ − ⟨ y ⁡ ( t ) ​ y ​ ( t + 1 ) ⟩ \langle y^{2}\rangle-\langle y(t)y(t+1)\rangle , so that for unit-variance outputs, minimizing Δ \Delta is equivalent to maximizing the temporal correlation ⟨ y ⁡ ( t ) ​ y ​ ( t + 1 ) ⟩ \langle y(t)y(t+1)\rangle . The SFA constraints are therefore precisely whitening, and the SFA objective is precisely the alignment loss used in our setting.

##### Nonlinear identifiability via SFA.

Sprekeler et al. [72] analyzed SFA in the setting where the input data x ⁡ ( t ) = F ⁡ ( s ⁡ ( t ) ) x(t)=F(s(t)) are generated from statistically independent latent variables s ⁡ ( t ) s(t) via an unknown invertible nonlinear mixing function F F . They showed that if the function space ℱ \mathcal{F} accessible to SFA is unrestricted (a Sobolev space), then the set of achievable output signals is independent of F F : for every g ∈ ℱ g\in\mathcal{F} , the composition g ∘ F g\circ F is also in ℱ \mathcal{F} , so the mixing function drops out of the analysis entirely. This is the same role played by our assumption that the encoder is expressive enough to invert the mixing function.

##### Eigenfunction structure and Sturm–Liouville theory.

Building on Franzius et al. [90] , Sprekeler et al. [72] showed that the optimal SFA functions satisfy an eigenvalue equation 𝒟 ​ g = λ ​ g \mathcal{D}g=\lambda g for a second-order partial differential operator 𝒟 \mathcal{D} . When the latent variables are statistically independent, the operator separates as 𝒟 = ∑ α 𝒟 α \mathcal{D}=\sum_{\alpha}\mathcal{D}_{\alpha} , where each 𝒟 α \mathcal{D}_{\alpha} depends on a single latent variable s α s_{\alpha} only. By separation of variables, the eigenfunctions of the full operator are products g 𝐢 ​ ( s ) = ∏ α g α ​ i α ​ ( s α ) g_{\mathbf{i}}(s)=\prod_{\alpha}g_{\alpha i_{\alpha}}(s_{\alpha}) of the eigenfunctions of the individual operators, with eigenvalues λ 𝐢 = ∑ α λ α ​ i α \lambda_{\mathbf{i}}=\sum_{\alpha}\lambda_{\alpha i_{\alpha}} . Each one-dimensional eigenvalue problem is of Sturm–Liouville type, and standard results guarantee that the eigenfunctions g α ​ i g_{\alpha i} are oscillatory with exactly i i zeros. In particular, the first non-constant eigenfunction g α ​ 1 g_{\alpha 1} is monotonic and therefore invertible, yielding identifiability of each latent variable up to a monotonic transformation for any latent variable distribution.

##### The Gaussian specialization and Hermite polynomials.

When the latent variables are reversible Gaussian processes (equivalently, Ornstein–Uhlenbeck processes), the conditional variance of the derivative given the current value is constant ( K α ​ ( s α ) = K α K_{\alpha}(s_{\alpha})=K_{\alpha} ), and the latent variable density is Gaussian. The Sturm–Liouville equation then reduces to Hermite’s differential equation, with eigenfunctions given by the Hermite polynomials H i H_{i} and eigenvalues λ α ​ i = i ​ K α \lambda_{\alpha i}=iK_{\alpha} . The first non-constant eigenfunction is simply g α ​ 1 ​ ( s α ) = s α g_{\alpha 1}(s_{\alpha})=s_{\alpha} : the identity. This is the same Hermite eigenstructure that appears in our proof via Mehler’s formula. Indeed, Mehler’s formula is the spectral expansion of the Green’s function (transition kernel) of the operator 𝒟 α \mathcal{D}_{\alpha} in the Gaussian case, so the two proof routes, Sturm–Liouville eigenvalue analysis and direct Mehler expansion, access the same underlying mathematical structure from different entry points.

##### Sequential versus simultaneous extraction.

A key difference between the approach of Sprekeler et al. [72] and ours lies in how the d d output dimensions are determined. Their xSFA algorithm extracts latent variables sequentially : it identifies the slowest latent variable, projects out all its nonlinear transformations, and repeats. This greedy procedure, combined with the assumption that the latent variables have distinct autocorrelation rates ( K α K_{\alpha} all different), yields identifiability up to permutation , a stronger result than our orthogonal identifiability. However, the sequential approach is fragile in practice: it requires representing the nonlinear transformations of already-extracted latent variables (approximated by high-degree polynomial expansions), accumulates estimation errors across iterations, and degrades rapidly as the number of latent variables increases [ 72 , Section 5] .

Our approach instead optimizes all d d output dimensions simultaneously via a joint loss (alignment plus whitening or SIGReg). Because whitening is invariant under orthogonal transformations, the simultaneous approach cannot resolve individual latent variable axes, yielding identifiability up to orthogonal rotation rather than permutation. This is the natural and complete identifiability class for the simultaneous setting.

##### The role of isotropic transitions.

The distinction between sequential and simultaneous extraction is intimately connected to the structure of the latent transitions. When the autocorrelation parameters ρ α \rho_{\alpha} differ across dimensions, the eigenvalue λ α ​ i = i ​ K α \lambda_{\alpha i}=iK_{\alpha} of the i i -th Hermite component of latent variable α \alpha depends on α \alpha . For the simultaneous approach to correctly identify the subspace of first harmonics (degree-1 Hermite components), all first harmonics must have smaller eigenvalues than all higher-order terms. This requires max α ⁡ K α < 2 ​ min β ​ K β \max_{\alpha}K_{\alpha}<2\min_{\beta}K_{\beta} , or equivalently, the fastest latent variable must not be more than twice as fast as the slowest. When ρ α \rho_{\alpha} are all equal (isotropic transitions), this condition is trivially satisfied, and the first d d eigenfunctions are exactly the d d first harmonics, yielding clean orthogonal identifiability.

We verified empirically that anisotropic transitions break the simultaneous approach: with distinct ρ α \rho_{\alpha} , the encoder recovers the second Hermite polynomial of the slow latent variable in place of the first Hermite polynomial of the fast latent variable, exactly as the eigenvalue interleaving predicts. See also the anisotropy in Tab. 2 . This confirms that isotropic transitions are not merely a simplifying assumption but a necessary condition for simultaneous (parallel) identifiability. The anisotropic case requires either sequential extraction à la Sprekeler et al. [72] or a modified objective such as the AnInfoNCE loss of Rusak et al. [91] , which reweights the contrastive objective to account for per-dimension variance differences, at the cost of downstream accuracy.

##### Beyond Gaussian latents.

The Sturm–Liouville perspective of Sprekeler et al. [72] suggests a generalization of our result that does not require Gaussian latents. If all latent variables are drawn i.i.d. from the same (arbitrary) distribution with the same transition structure, then the eigenvalue spectra of the individual operators 𝒟 α \mathcal{D}_{\alpha} are identical: λ α ​ i = λ i \lambda_{\alpha i}=\lambda_{i} for all α \alpha . All first harmonics share the same eigenvalue λ 1 \lambda_{1} , all second harmonics share λ 2 > λ 1 \lambda_{2}>\lambda_{1} (the gap is guaranteed by Sturm–Liouville theory), and the top- d d eigenspace of the simultaneous problem is exactly the span of the first harmonics. The simultaneous approach then yields identifiability up to orthogonal rotation, not in the original latent variable space but in the space of first harmonics. Since the first harmonics are monotonic functions of the latent variables, this amounts to identifiability up to an orthogonal rotation composed with a shared monotonic nonlinearity applied elementwise. Only in the Gaussian case does the first harmonic equal the identity, collapsing this to linear orthogonal identifiability. A rigorous treatment of this non-Gaussian generalization is an interesting direction for future work.

##### Relation to Mehler’s formula and maximal correlation.