LeJEPA: Provable and Scalable Self-Supervised Learning Without the Heuristics — part 8/10

Cleaned-text code points [103101, 117672). Bibliography omitted here; source.md includes it.

Our proof will follow the same proof as for Section B.8 . Necessity is immediate: if X = d Y X\stackrel{{\scriptstyle d}}{{=}}Y , then every measurable function of X X has the same distribution as the corresponding function of Y Y , from which the linear mapping x ↦ ⟨ u , x ⟩ x\mapsto\langle u,x\rangle for u ∈ 𝕊 d − 1 u\in\mathbb{S}^{d-1} is a special case. For sufficiency, assume ⟨ u , X ⟩ = d ⟨ u , Y ⟩ \langle u,X\rangle\stackrel{{\scriptstyle d}}{{=}}\langle u,Y\rangle for all u ∈ 𝕊 d − 1 u\in\mathbb{S}^{d-1} . Let φ X ​ ( t ) := 𝔼 ⁡ [ e i ​ ⟨ t , X ⟩ ] \varphi_{X}(t):=\mathbb{E}\big[e^{i\langle t,X\rangle}\big] and φ Y ​ ( t ) := 𝔼 ⁡ [ e i ​ ⟨ t , Y ⟩ ] \varphi_{Y}(t):=\mathbb{E}\big[e^{i\langle t,Y\rangle}\big] denote the characteristic functions of X X and Y Y . Fix an arbitrary t ∈ ℝ d t\in\mathbb{R}^{d} ; if t = 0 t=0 , then φ X ​ ( 0 ) = φ Y ​ ( 0 ) = 1 \varphi_{X}(0)=\varphi_{Y}(0)=1 . If t ≠ 0 t\neq 0 , write t = s ​ u t=su with s := ‖ t ‖ > 0 s:=\|t\|>0 and u := t / ‖ t ‖ ∈ 𝕊 d − 1 u:=t/\|t\|\in\mathbb{S}^{d-1} . By the assumption, ⟨ u , X ⟩ = d ⟨ u , Y ⟩ \langle u,X\rangle\stackrel{{\scriptstyle d}}{{=}}\langle u,Y\rangle , hence for this u u and s s we have φ X ​ ( t ) = 𝔼 ⁡ [ e i ​ ⟨ t , X ⟩ ] = 𝔼 ⁡ [ e i ​ s ​ ⟨ u , X ⟩ ] = 𝔼 ⁡ [ e i ​ s ​ ⟨ u , Y ⟩ ] = 𝔼 ⁡ [ e i ​ ⟨ t , Y ⟩ ] = φ Y ​ ( t ) . \varphi_{X}(t)\;=\;\mathbb{E}\big[e^{i\langle t,X\rangle}\big]\;=\;\mathbb{E}\big[e^{is\langle u,X\rangle}\big]\;=\;\mathbb{E}\big[e^{is\langle u,Y\rangle}\big]\;=\;\mathbb{E}\big[e^{i\langle t,Y\rangle}\big]\;=\;\varphi_{Y}(t). Thus φ X ​ ( t ) = φ Y ​ ( t ) \varphi_{X}(t)=\varphi_{Y}(t) for all t ∈ ℝ d t\in\mathbb{R}^{d} , i.e., φ X ≡ φ Y \varphi_{X}\equiv\varphi_{Y} on ℝ d \mathbb{R}^{d} . By the uniqueness theorem for characteristic functions, this implies X = d Y X\stackrel{{\scriptstyle d}}{{=}}Y . (ii) Define ψ n , t := 𝔼 ⁡ [ e i ​ ⟨ t , X n ⟩ ] \psi_{n,t}:=\mathbb{E}\big[e^{i\langle t,X_{n}\rangle}\big] and ψ t := 𝔼 ⁡ [ e i ​ ⟨ t , X ⟩ ] \psi_{t}:=\mathbb{E}\big[e^{i\langle t,X\rangle}\big] . Fix t ∈ ℝ d t\in\mathbb{R}^{d} and decompose t = s ​ u t=su with s := ‖ t ‖ ≥ 0 s:=\|t\|\geq 0 and u ∈ 𝕊 d − 1 u\in\mathbb{S}^{d-1} (take, e.g., u = t / ‖ t ‖ u=t/\|t\| if t ≠ 0 t\neq 0 , and any u u if t = 0 t=0 ). The map g s : ℝ → ℝ g_{s}:\mathbb{R}\to\mathbb{R} , g s ​ ( x ) = s ​ x g_{s}(x)=sx , is continuous. By the continuous mapping theorem applied to the real-valued random variables ⟨ u , X n ⟩ → 𝑑 ⟨ u , X ⟩ \langle u,X_{n}\rangle\xrightarrow{d}\langle u,X\rangle , we obtain ⟨ t , X n ⟩ = s ⁡ ⟨ u , X n ⟩ → 𝑑 s ⁡ ⟨ u , X ⟩ = ⟨ t , X ⟩ . \langle t,X_{n}\rangle\;=\;s\langle u,X_{n}\rangle\xrightarrow{d}s\langle u,X\rangle\;=\;\langle t,X\rangle. Hence, for every fixed t ∈ ℝ d t\in\mathbb{R}^{d} , the one-dimensional projections satisfy ⟨ t , X n ⟩ → 𝑑 ⟨ t , X ⟩ \langle t,X_{n}\rangle\xrightarrow{d}\langle t,X\rangle , which in turn yields pointwise convergence of characteristic functions: ψ n , t = 𝔼 ⁡ [ e i ​ ⟨ t , X n ⟩ ] ⟶ 𝔼 ⁡ [ e i ​ ⟨ t , X ⟩ ] = ψ t , for all ​ t ∈ ℝ d . \psi_{n,t}\;=\;\mathbb{E}\big[e^{i\langle t,X_{n}\rangle}\big]\;\longrightarrow\;\mathbb{E}\big[e^{i\langle t,X\rangle}\big]\;=\;\psi_{t},\qquad\text{for all }t\in\mathbb{R}^{d}. Therefore, by Lévy’s continuity theorem, X n → 𝑑 X X_{n}\xrightarrow{d}X . This completes the proof. ∎

### B.9 Proof of Section 4.1

###### Proof.

We first formulate the following assumptions required for the proof–all of this are satisfied by typical univariate statistical tests.

P = Q P=Q if and only if P a = Q a P_{a}=Q_{a} for all a ∈ S d − 1 a\in S^{d-1} (population-level equivalence of laws).

A n A_{n} are finite sets with mesh Δ ⁡ ( A n ) := sup u ∈ S d − 1 min a ∈ A n ⁡ ‖ u − a ‖ → 0 \Delta(A_{n}):=\sup_{u\in S^{d-1}}\min_{a\in A_{n}}\|u-a\|\to 0 as n → ∞ n\to\infty .

If P ≠ Q P\neq Q , there exists a separating direction a ⋆ ∈ S d − 1 a^{\star}\in S^{d-1} and a neighborhood U U of a ⋆ a^{\star} such that inf a ∈ U lim n → ∞ Pr ⁡ ( T a , n ≥ u n ​ ( α ) ) = 1 . \inf_{a\in U}\lim_{n\to\infty}\Pr\big(T_{a,n}\geq u_{n}(\alpha)\big)=1. (Intuitively: near a truly separating direction, the 1D statistic eventually exceeds the global null threshold with probability → 1 \to 1 .)

(i) Under H 0 : P = Q H_{0}:P=Q , our assumption implies no separating direction exists at the population level, and the calibration of u n ​ ( α ) u_{n}(\alpha) ensures Pr ⁡ ( M n ≥ u n ​ ( α ) ) ≤ α \Pr(M_{n}\geq u_{n}(\alpha))\leq\alpha for all n n , hence lim sup n → ∞ Pr ⁡ ( Ψ n = 1 ) ≤ α \limsup_{n\to\infty}\Pr(\Psi_{n}=1)\leq\alpha . (ii) Suppose P ≠ Q P\neq Q . Our assumption guarantees that there exists at least one separating direction a ⋆ a^{\star} with P a ⋆ ≠ Q a ⋆ P_{a^{\star}}\neq Q_{a^{\star}} . Our assumption guarantees a neighborhood U U of a ⋆ a^{\star} in which the projection statistics exceed the global null threshold with probability tending to 1: inf a ∈ U lim n → ∞ Pr ⁡ ( T a , n ≥ u n ​ ( α ) ) = 1 . \inf_{a\in U}\lim_{n\to\infty}\Pr\big(T_{a,n}\geq u_{n}(\alpha)\big)\;=\;1. By assumption, for all large n n the set A n A_{n} contains at least one direction a n ∈ U a_{n}\in U (dense coverage). Therefore, Pr ⁡ ( Ψ n = 1 ) = Pr ⁡ ( M n ≥ u n ​ ( α ) ) ≥ Pr ⁡ ( T a n , n ≥ u n ​ ( α ) ) ⟶ 1 , \Pr(\Psi_{n}=1)\;=\;\Pr\big(M_{n}\geq u_{n}(\alpha)\big)\;\geq\;\Pr\big(T_{a_{n},n}\geq u_{n}(\alpha)\big)\;\longrightarrow\;1, which proves consistency. ∎

### B.10 Proof of Section 4.3

###### Proof.

For each case, consider the function g ⁡ ( a ) g(a) on 𝕊 D − 1 \mathbb{S}^{D-1} defined by the quantity of interest (CF, CDF, or moment) at a fixed t t or k k . Since f ∈ H α ​ ( ℝ D ) f\in H^{\alpha}(\mathbb{R}^{D}) , the mapping a ↦ g ⁡ ( a ) a\mapsto g(a) is in H α ​ ( 𝕊 D − 1 ) H^{\alpha}(\mathbb{S}^{D-1}) for each fixed t t or k k .

Given M M samples { a i } i = 1 M \{a_{i}\}_{i=1}^{M} on the sphere, the best possible reconstruction of g g from its values at these points is given by spherical interpolation. By classical results on Sobolev spaces and spherical harmonics (see, e.g., Narcowich et al. [2006] ), the L 2 L^{2} interpolation error for functions in H α ​ ( 𝕊 D − 1 ) H^{\alpha}(\mathbb{S}^{D-1}) using M M points is bounded by 𝔼 b [ | g ( b ) − g ∗ ( b ) | 2 ] ≤ C ( D , α ) M − 2 α / ( D − 1 ) ∥ g ∥ H α ​ ( 𝕊 D − 1 ) 2 , \mathbb{E}_{b}\left[|g(b)-g^{*}(b)|^{2}\right]\leq C(D,\alpha)M^{-2\alpha/(D-1)}\|g\|_{H^{\alpha}(\mathbb{S}^{D-1})}^{2}, where g ∗ g^{*} is the interpolant matching g g at the M M sampled points. The interpolation error bound on the sphere follows from the theory of spherical harmonics and Marcinkiewicz–Zygmund (MZ) inequalities . Any f ∈ H α ​ ( 𝕊 d ) f\in H^{\alpha}(\mathbb{S}^{d}) admits a spherical harmonics expansion, and the best L 2 L^{2} approximation by harmonics of degree at most L L satisfies ∥ f − P L f ∥ L 2 ​ ( 𝕊 d ) ≤ ( 1 + L 2 ) − α / 2 ∥ f ∥ H α ​ ( 𝕊 d ) , \|f-P_{L}f\|_{L^{2}(\mathbb{S}^{d})}\leq(1+L^{2})^{-\alpha/2}\|f\|_{H^{\alpha}(\mathbb{S}^{d})}, where P L ​ f P_{L}f is the projection onto harmonics of degree ≤ L \leq L [ Narcowich et al., 2006 , Lemma 2.1] . If M M points are distributed quasi-uniformly on 𝕊 d \mathbb{S}^{d} , then for L ∼ c ​ M 1 / d L\sim cM^{1/d} , the set forms a Marcinkiewicz–Zygmund (MZ) set for degree L L [ Mhaskar et al., 2001 , Theorem 1.1] . This allows reconstruction of any function in the space of harmonics of degree at most L L from its values at these points, and the L 2 L^{2} interpolation error for f f is bounded by ∥ f − I M f ∥ L 2 ​ ( 𝕊 d ) ≤ C ( 1 + L 2 ) − α / 2 ∥ f ∥ H α ​ ( 𝕊 d ) , \|f-I_{M}f\|_{L^{2}(\mathbb{S}^{d})}\leq C(1+L^{2})^{-\alpha/2}\|f\|_{H^{\alpha}(\mathbb{S}^{d})}, where I M ​ f I_{M}f is any interpolant matching f f at the M M points [ Narcowich et al., 2006 , Theorem 3.1] . Substituting L ∼ c ​ M 1 / d L\sim cM^{1/d} yields the rate M − α / d M^{-\alpha/d} , and thus 𝔼 ω | f ( ω ) − I M f ( ω ) | 2 ≤ C ( d , α ) M − 2 α / d ∥ f ∥ H α ​ ( 𝕊 d ) 2 , \mathbb{E}_{\omega}|f(\omega)-I_{M}f(\omega)|^{2}\leq C(d,\alpha)M^{-2\alpha/d}\|f\|_{H^{\alpha}(\mathbb{S}^{d})}^{2}, with explicit C ⁡ ( d , α ) C(d,\alpha) as in the main theorem. Integrating (or summing) over t t (for CF and CDF) or k k (for moments, with weights w k w_{k} ) yields the stated bounds. The explicit constant C ⁡ ( D , α ) C(D,\alpha) arises from the theory of spherical Sobolev spaces and is given above.

For the moment case, the sum over k k is weighted to ensure convergence, as higher moments may grow rapidly. The weights w k w_{k} can be chosen, for example, as w k = 1 / k ! w_{k}=1/k! .

This completes the proof. ∎

### B.11 Proof of Section 4.2.1

Pick distinct x 0 , … , x K + 1 ∈ ℝ x_{0},\dots,x_{K+1}\in\mathbb{R} and consider the linear map A : ℝ K + 2 → ℝ K + 1 A:\mathbb{R}^{K+2}\to\mathbb{R}^{K+1} , ( A ​ p ) r = ∑ j = 0 K + 1 p j ​ x j r (Ap)_{r}=\sum_{j=0}^{K+1}p_{j}x_{j}^{r} for r = 0 , … , K r=0,\dots,K . Then rank ⁡ ( A ) ≤ K + 1 \mathrm{rank}(A)\leq K+1 , so ker ⁡ ( A ) ≠ { 0 } \ker(A)\neq\{0\} . Let v ∈ ker ⁡ ( A ) ∖ { 0 } v\in\ker(A)\setminus\{0\} ; from ( A ​ p ) 0 = ∑ j p j (Ap)_{0}=\sum_{j}p_{j} , we get ∑ j v j = 0 \sum_{j}v_{j}=0 , hence v v has positive and negative entries. Choose a strictly positive probability vector p p and ε > 0 \varepsilon>0 small such that p ± := p ± ε ​ v p^{\pm}:=p\pm\varepsilon v remain probability vectors. Then A ​ p + = A ​ p − Ap^{+}=Ap^{-} , so the distributions supported on { x j } \{x_{j}\} with masses p ± p^{\pm} are distinct yet match moments up to order K K .

### B.12 Proof of Section 4.2.3

###### Proof.

Fix the Gaussian weight w s ​ ( t ) = e − s 2 ​ t 2 , s > 0 , w_{s}(t)=e^{-s^{2}t^{2}},\qquad s>0, and define the population CF distance D ⁡ ( P , G ) = ∫ ℝ w s ​ ( t ) ​ | φ P ​ ( t ) − φ G ​ ( t ) | 2 ​ 𝑑 t . D(P,G)=\int_{\mathbb{R}}w_{s}(t)\big|\varphi_{P}(t)-\varphi_{G}(t)\big|^{2}dt. Let the empirical CF be φ ^ N ​ ( t ) = 1 N ​ ∑ i = 1 N e i ​ t ​ X i , \widehat{\varphi}_{N}(t)=\frac{1}{N}\sum_{i=1}^{N}e^{itX_{i}}, and consider the V-statistic estimator D ^ V = ∫ ℝ w s ​ ( t ) ​ | φ ^ N ​ ( t ) − φ G ​ ( t ) | 2 ​ 𝑑 t . \widehat{D}_{V}=\int_{\mathbb{R}}w_{s}(t)\big|\widehat{\varphi}_{N}(t)-\varphi_{G}(t)\big|^{2}dt. We use only that | e i ​ t ​ X | = 1 |e^{itX}|=1 , | φ P ​ ( t ) | ≤ 1 |\varphi_{P}(t)|\leq 1 , | φ G ​ ( t ) | ≤ 1 |\varphi_{G}(t)|\leq 1 , and integrability of w s w_{s} . For each i i differentiate under the integral (dominated convergence applies because the integrand and its derivative are bounded) ∂ D ^ V ∂ X i = \displaystyle\frac{\partial\widehat{D}_{V}}{\partial X_{i}}= ∫ ℝ w s ​ ( t ) ​ 2 ​ ℜ ⁡ ( ( φ ^ N ​ ( t ) − φ G ​ ( t ) ) ​ ∂ φ ^ N ​ ( t ) ∂ X i ¯ ) ​ 𝑑 t , \displaystyle\int_{\mathbb{R}}w_{s}(t)2\Re\!\Big(\big(\widehat{\varphi}_{N}(t)-\varphi_{G}(t)\big)\overline{\frac{\partial\widehat{\varphi}_{N}(t)}{\partial X_{i}}}\Big)dt, ∂ φ ^ N ​ ( t ) ∂ X i = \displaystyle\frac{\partial\widehat{\varphi}_{N}(t)}{\partial X_{i}}= 1 N ​ i ​ t ​ e i ​ t ​ X i , \displaystyle\frac{1}{N}ite^{itX_{i}}, since | φ ^ N ​ ( t ) | ≤ 1 |\widehat{\varphi}_{N}(t)|\leq 1 and | φ G ​ ( t ) | ≤ 1 |\varphi_{G}(t)|\leq 1 , | ∂ D ^ V ∂ X i | \displaystyle\left|\frac{\partial\widehat{D}_{V}}{\partial X_{i}}\right| ≤ 2 N ​ ∫ w s ​ ( t ) ​ | t | ​ ( | φ ^ N ​ ( t ) | + | φ G ​ ( t ) | ) ​ 𝑑 t \displaystyle\leq\frac{2}{N}\int w_{s}(t)|t|\big(|\widehat{\varphi}_{N}(t)|+|\varphi_{G}(t)|\big)dt ≤ 4 N ​ ∫ w s ​ ( t ) ​ | t | ​ 𝑑 t \displaystyle\leq\frac{4}{N}\int w_{s}(t)|t|dt = 4 N ​ s 2 , \displaystyle=\frac{4}{Ns^{2}}, using ∫ ℝ e − s 2 ​ t 2 ​ | t | ​ 𝑑 t = 1 / s 2 \int_{\mathbb{R}}e^{-s^{2}t^{2}}|t|dt=1/s^{2} . | ∂ D ^ V ∂ X i | ≤ 4 N ​ ∫ ℝ w s ​ ( t ) ​ | t | ​ 𝑑 t = 4 N ​ s 2 . \Bigg|\frac{\partial\widehat{D}_{V}}{\partial X_{i}}\Bigg|\;\leq\;\frac{4}{N}\int_{\mathbb{R}}w_{s}(t)|t|dt\;=\;\frac{4}{Ns^{2}}. Moreover, differentiating once more in X i X_{i} and using | φ ^ N ​ ( t ) | ≤ 1 |\widehat{\varphi}_{N}(t)|\leq 1 , | φ G ​ ( t ) | ≤ 1 |\varphi_{G}(t)|\leq 1 gives a global Lipschitz bound | ∂ 2 D ^ V ∂ X i 2 | ≤ C N ​ ∫ ℝ w s ​ ( t ) ​ t 2 ​ 𝑑 t = C N ⋅ π 2 ​ s 3 , \Bigg|\frac{\partial^{2}\widehat{D}_{V}}{\partial X_{i}^{2}}\Bigg|\;\leq\;\frac{C}{N}\int_{\mathbb{R}}w_{s}(t)t^{2}dt\;=\;\frac{C}{N}\cdot\frac{\sqrt{\pi}}{2s^{3}}, for some absolute constant C C arising from bounded factors and product rule. Hence ECF gradients are uniformly bounded and Lipschitz, with scale controlled only by ( N , s ) (N,s) .

(Moment sample-gradients are polynomial in X i X_{i} and unbounded for k ≥ 2 k\geq 2 .) Let D ^ V \widehat{D}_{V} be as above. Define the moment objective D ^ k = ( ϕ ¯ − μ ) ⊤ ​ W ​ ( ϕ ¯ − μ ) , ϕ ¯ := 1 N ​ ∑ i = 1 N ϕ ⁡ ( X i ) , ϕ ⁡ ( x ) = ( x , x 2 , … , x k ) ⊤ , \widehat{D}_{k}\;=\;(\bar{\phi}-\mu)^{\top}W(\bar{\phi}-\mu),\qquad\bar{\phi}:=\frac{1}{N}\sum_{i=1}^{N}\phi(X_{i}),\quad\phi(x)=(x,x^{2},\dots,x^{k})^{\top}, for a symmetric positive semidefinite W ∈ ℝ k × k W\in\mathbb{R}^{k\times k} and Gaussian target moments μ = 𝔼 G ​ [ ϕ ​ ( Y ) ] \mu=\mathbb{E}_{G}[\phi(Y)] . For each i i , ∂ D ^ k ∂ X i = \displaystyle\frac{\partial\widehat{D}_{k}}{\partial X_{i}}= 2 N ​ ( ϕ ¯ − μ ) ⊤ ​ W ​ ∂ ϕ ⁡ ( X i ) ∂ X i , \displaystyle\frac{2}{N}(\bar{\phi}-\mu)^{\top}W\frac{\partial\phi(X_{i})}{\partial X_{i}}, ∂ ϕ ⁡ ( X ) ∂ X = \displaystyle\frac{\partial\phi(X)}{\partial X}= ( 1 , 2 ​ X , 3 ​ X 2 , … , k ​ X k − 1 ) ⊤ . \displaystyle\big(1,2X,3X^{2},\dots,kX^{k-1}\big)^{\top}. The gradient formula follows by the chain rule and linearity of ϕ ¯ \bar{\phi} . Let c := W ⁡ ( ϕ ¯ − μ ) c:=W(\bar{\phi}-\mu) and write c r c_{r} for its r r -th coordinate. Then ∂ D ^ k ∂ X i = 2 N ​ ∑ r = 1 k c r ​ r ​ X i r − 1 , \frac{\partial\widehat{D}_{k}}{\partial X_{i}}=\frac{2}{N}\sum_{r=1}^{k}c_{r}rX_{i}^{r-1}, which is a polynomial in X i X_{i} of degree deg = max ⁡ { r − 1 : c r ≠ 0 } ≤ k − 1 \deg=\max\{r-1:c_{r}\neq 0\}\leq k-1 . In particular, if c k ≠ 0 c_{k}\neq 0 (the generic case when the top-weighted deviation is nonzero), then | ∂ D ^ k ∂ X i | → | X i | → ∞ ∞ as | X i | k − 1 . \left|\frac{\partial\widehat{D}_{k}}{\partial X_{i}}\right|\;\xrightarrow[|X_{i}|\to\infty]{}\;\infty\quad\text{as}\quad|X_{i}|^{k-1}. The expression is a nonconstant polynomial in X i X_{i} of degree deg ≤ k − 1 \deg\leq k-1 whenever some c r ≠ 0 c_{r}\neq 0 with r ≥ 2 r\geq 2 . Thus the gradient cannot be uniformly bounded on ℝ \mathbb{R} . If c k ≠ 0 c_{k}\neq 0 , the leading term dominates and the magnitude grows like | X i | k − 1 |X_{i}|^{k-1} , proving unboundedness for k ≥ 2 k\geq 2 . ∎

### B.13 Proof of Section 5.1

###### Proof.