LeJEPA: Provable and Scalable Self-Supervised Learning Without the Heuristics — part 10/10

Cleaned-text code points [132698, 139605). Bibliography omitted here; source.md includes it.

We ideally would like to compare the distributions. One slight variation is to compare the Characteristic function of the distributions. Given samples 𝒙 1 , … , 𝒙 N {\bm{x}}_{1},\dots,{\bm{x}}_{N} , the Empirical Characteristic Function (ECF) is defined as ψ ^ N ​ ( 𝒕 ) = 1 N ​ ∑ n = 1 N e − i ​ 𝒕 ⊤ ​ 𝒚 n . \displaystyle\hat{\psi}_{N}({\bm{t}})=\frac{1}{N}\sum_{n=1}^{N}e^{-i{\bm{t}}^{\top}{\bm{y}}_{n}}. We can now compare our ECF to the one of the target distribution and build the statistic N ∫ | ψ ^ N ( 𝒕 ) − ψ 0 ( 𝒕 ) | 2 ω ( 𝒕 ) d t = N ∫ | ψ ^ N ( 𝒕 ) − e − ∥ 𝒕 ∥ 2 / 2 | 2 ω ( 𝒕 ) d t , \displaystyle N\int|\hat{\psi}_{N}({\bm{t}})-\psi_{0}({\bm{t}})|^{2}\omega({\bm{t}})dt=N\int|\hat{\psi}_{N}({\bm{t}})-e^{-\|{\bm{t}}\|_{2}/2}|^{2}\omega({\bm{t}})dt, if the weighting function is given by ω ( 𝒕 ) = ( 2 π β 2 ) − d / 2 e − ‖ 𝒕 ‖ 2 2 2 \omega({\bm{t}})=(2\pi\beta^{2})^{-d/2}e^{-\frac{\|{\bm{t}}\|_{2}^{2}}{2}} then the following simplification can be made BHEP n , β = \displaystyle\mathrm{BHEP}_{n,\beta}= 1 n ​ ∑ j , k = 1 n exp ⁡ ( − β 2 ​ ‖ Y n , j − Y n , k ‖ 2 2 ) \displaystyle\frac{1}{n}\sum_{j,k=1}^{n}\exp\left(-\frac{\beta^{2}\left\|Y_{n,j}-Y_{n,k}\right\|^{2}}{2}\right) − 2 ( 1 + β 2 ) d / 2 ∑ j = 1 n exp ( − β 2 ​ ‖ Y n , j ‖ 2 2 ​ ( 1 + β 2 ) ) + n ( 1 + 2 ​ β 2 ) d / 2 . \displaystyle-\frac{2}{\left(1+\beta^{2}\right)^{d/2}}\sum_{j=1}^{n}\exp\left(-\frac{\beta^{2}\left\|Y_{n,j}\right\|^{2}}{2\left(1+\beta^{2}\right)}\right)+\frac{n}{\left(1+2\beta^{2}\right)^{d/2}}. with β > 0 \beta>0 , Baringhaus-Henze-Epps-Pulley. From 1 1 1 https://www.routledge.com/Density-Estimation-for-Statistics-and-Data-Analysis/Silverman/p/book/9780412246203?srsltid=AfmBOoodlL-CtlqL0JVC-LcP6mOWw6VTt51_YstdZOW4W3iuicu1VFyg leading to the HZ test 2 2 2 https://www.tandfonline.com/doi/abs/10.1080/03610929008830400 uses β n = 2 − 1 / 2 ( ( 2 d + 1 ) n / 4 ) 1 / ( d + 4 ) \beta_{n}=2^{-1/2}((2d+1)n/4)^{1/(d+4)} (59)

the same can be done with the moment generating function 3 3 3 https://arxiv.org/pdf/1711.07199 T n , β = π d / 2 ​ ( 1 n ​ ∑ i , j = 1 n 1 β d / 2 ​ exp ⁡ ( ‖ Y n , i + Y n , j ‖ 2 4 ​ β ) + n ( β − 1 ) d / 2 CLOSE \displaystyle T_{n,\beta}=\pi^{d/2}\left(\frac{1}{n}\sum_{i,j=1}^{n}\frac{1}{\beta^{d/2}}\exp\left(\frac{\left\|Y_{n,i}+Y_{n,j}\right\|^{2}}{4\beta}\right)+\frac{n}{(\beta-1)^{d/2}}\right. − 2 ∑ j = 1 n 1 ( β − 1 / 2 ) d / 2 exp ( ‖ Y n , j ‖ 2 4 ​ β − 2 ) ) , \displaystyle\left.-2\sum_{j=1}^{n}\frac{1}{(\beta-1/2)^{d/2}}\exp\left(\frac{\left\|Y_{n,j}\right\|^{2}}{4\beta-2}\right)\right), here with β > 2 \beta>2

There is also one combining both 4 4 4 https://arxiv.org/pdf/1706.03029 ! T n , γ := ∫ ℝ d U n 2 ​ ( t ) ​ w γ ​ ( t ) ​ 𝑑 t U n ​ ( t ) := n ​ ( R n ​ ( t ) ​ M n ​ ( t ) − 1 ) \begin{array}[]{l}T_{n,\gamma}:=\int_{\mathbb{R}^{d}}U_{n}^{2}(t)w_{\gamma}(t)\mathrm{d}t\\ U_{n}(t):=\sqrt{n}\left(R_{n}(t)M_{n}(t)-1\right)\end{array} (60) T n , γ = \displaystyle T_{n,\gamma}= ( π γ ) d / 2 { 1 2 ​ n 3 ∑ j , k , l , m = 1 n [ exp ( ‖ Y j ​ k + ‖ 2 − ‖ Y ℓ ​ m − ‖ 2 4 ​ γ ) cos ( Y j ​ k + ⁣ ⊤ ​ Y ℓ ​ m − 2 ​ γ ) \displaystyle\left(\frac{\pi}{\gamma}\right)^{d/2}\left\{\frac{1}{2n^{3}}\sum_{j,k,l,m=1}^{n}\left[\exp\left(\frac{\left\|Y_{jk}^{+}\right\|^{2}-\left\|Y_{\ell m}^{-}\right\|^{2}}{4\gamma}\right)\cos\left(\frac{Y_{jk}^{+\top}Y_{\ell m}^{-}}{2\gamma}\right)\right.\right. (61) + \displaystyle+ exp ( ‖ Y j ​ k + ‖ 2 − ‖ Y ℓ ​ m + ‖ 2 4 ​ γ ) cos ( Y j ​ k + ⁣ ⊤ ​ Y ℓ ​ m + 2 ​ γ ) ] \displaystyle\left.\exp\left(\frac{\left\|Y_{jk}^{+}\right\|^{2}-\left\|Y_{\ell m}^{+}\right\|^{2}}{4\gamma}\right)\cos\left(\frac{Y_{jk}^{+\top}Y_{\ell m}^{+}}{2\gamma}\right)\right] − 2 n ∑ j , k = 1 n exp ( ‖ Y n , j ‖ 2 − ‖ Y n , k ‖ 2 4 ​ γ ) cos ( Y n , j ⊤ ​ Y n , k 2 ​ γ ) + n } , \displaystyle\left.-\frac{2}{n}\sum_{j,k=1}^{n}\exp\left(\frac{\left\|Y_{n,j}\right\|^{2}-\left\|Y_{n,k}\right\|^{2}}{4\gamma}\right)\cos\left(\frac{Y_{n,j}^{\top}Y_{n,k}}{2\gamma}\right)+n\right\}, and its simplified version T ~ n , γ := ∫ ℝ d U n ​ ( t ) ​ w γ ​ ( t ) ​ 𝑑 t . \widetilde{T}_{n,\gamma}:=\int_{\mathbb{R}^{d}}U_{n}(t)w_{\gamma}(t)\mathrm{d}t. (62) T ~ n , γ = ( π γ ) d / 2 ​ n ​ ( 1 n 2 ​ ∑ j , k = 1 n exp ⁡ ( ‖ Y n , j ‖ 2 − ‖ Y n , k ‖ 2 4 ​ γ ) ​ cos ⁡ ( Y n , j ⊤ ​ Y n , k 2 ​ γ ) − 1 ) \widetilde{T}_{n,\gamma}=\left(\frac{\pi}{\gamma}\right)^{d/2}\sqrt{n}\left(\frac{1}{n^{2}}\sum_{j,k=1}^{n}\exp\left(\frac{\left\|Y_{n,j}\right\|^{2}-\left\|Y_{n,k}\right\|^{2}}{4\gamma}\right)\cos\left(\frac{Y_{n,j}^{\top}Y_{n,k}}{2\gamma}\right)-1\right) (63)

Also one testing the derivative 5 5 5 https://arxiv.org/pdf/1901.03986

HV n , γ := n ​ ∫ ‖ ∇ M n ​ ( t ) − t ​ M n ​ ( t ) ‖ 2 ​ w ~ γ ​ ( t ) ​ 𝑑 t \mathrm{HV}_{n,\gamma}:=n\int\left\|\nabla M_{n}(t)-tM_{n}(t)\right\|^{2}\widetilde{w}_{\gamma}(t)\mathrm{d}t (64) HV n , γ = 1 n ​ ( π γ ) d / 2 ​ ∑ j , k = 1 n exp ⁡ ( ‖ Y n , j , k + ‖ 2 4 ​ γ ) ​ ( Y n , j ⊤ ​ Y n , k − ‖ Y n , j , k + ‖ 2 2 ​ γ + d 2 ​ γ + ‖ Y n , j , k + ‖ 2 4 ​ γ 2 ) . \mathrm{HV}_{n,\gamma}=\frac{1}{n}\left(\frac{\pi}{\gamma}\right)^{d/2}\sum_{j,k=1}^{n}\exp\left(\frac{\left\|Y_{n,j,k}^{+}\right\|^{2}}{4\gamma}\right)\left(Y_{n,j}^{\top}Y_{n,k}-\frac{\left\|Y_{n,j,k}^{+}\right\|^{2}}{2\gamma}+\frac{d}{2\gamma}+\frac{\left\|Y_{n,j,k}^{+}\right\|^{2}}{4\gamma^{2}}\right). (65)

skewness 6 6 6 https://www.jstor.org/stable/2334770 : b 1 , d = 1 n 2 ​ ∑ j , k = 1 n ( Y n , j ⊤ ​ Y n , k ) 3 b_{1,d}=\frac{1}{n^{2}}\sum_{j,k=1}^{n}\left(Y_{n,j}^{\top}Y_{n,k}\right)^{3} (66) skewness 7 7 7 https://link.springer.com/article/10.1007/s13171-020-00211-6 : b ~ 1 , d = 1 n 2 ​ ∑ j , k = 1 n Y n , j ⊤ ​ Y n , k ​ ‖ Y n , j ‖ 2 ​ ‖ Y n , k ‖ 2 \widetilde{b}_{1,d}=\frac{1}{n^{2}}\sum_{j,k=1}^{n}Y_{n,j}^{\top}Y_{n,k}\left\|Y_{n,j}\right\|^{2}\left\|Y_{n,k}\right\|^{2} (67) which should be 0 for Gaussian and Kurtosis which should be d(d+2) b 2 , d = 1 n ​ ∑ j = 1 n ‖ Y n , j ‖ 4 b_{2,d}=\frac{1}{n}\sum_{j=1}^{n}\left\|Y_{n,j}\right\|^{4} (68)

## Instructions for reporting errors

We are continuing to improve HTML versions of papers, and your feedback helps enhance accessibility and mobile support. To report errors in the HTML that will help us improve conversion and rendering, choose any of the methods listed below:

Click the "Report Issue" ( ) button, located in the page header.

Tip: You can select the relevant text first, to include it in your report.

Our team has already identified the following issues . We appreciate your time reviewing and reporting rendering errors we may not have found yet. Your efforts will help us improve the HTML versions for all readers, because disability should not be a barrier to accessing research. Thank you for your continued support in championing open access for all.

Have a free development cycle? Help support accessibility at arXiv! Our collaborators at LaTeXML maintain a list of packages that need conversion , and welcome developer contributions .