LeJEPA: Provable and Scalable Self-Supervised Learning Without the Heuristics

arxiv:2511.08544

Original source · Paper JSON

Self-supervised learning and vision · 2025-11-11

Cites (6) · Cited by (2)

Complete cleaned Markdown · Source Markdown including bibliography

Read in parts

1. Part 1
2. Part 2
3. Part 3
4. Part 4
5. Part 5
6. Part 6
7. Part 7
8. Part 8
9. Part 9
10. Part 10

Contents

• 1 Introduction
• 2 Background and Notations
• 2.1 Notations and Definitions
• 2.2 The Need for Reliable Pretraining
• 2.3 The Need for Actionable Theory
• 3 Latent Euclidean: Embeddings Should be Isotropic Gaussian
• 3.1 Linear Probing
• 3.2 Nonlinear Probing
• 3.3 Geometric and Practical Insights
• 4 SIGReg: Reliable Isotropic Gaussian Regularization in High-Dimension
• 4.1 Hypothesis Testing as a Judge
• 4.2 SIGReg: Sketching the Epps-Pulley Test is Stable and Scalable
• 4.2.1 Moments are Unstable and Insufficient
• 4.2.2 Cumulative Density Functions are Impractical
• 4.2.3 Characteristic Functions are Stable, Scalable and Identifiable
• 4.3 How SIGReg Beats the Curse of Dimensionality
• Smoothness Beats the Curse of Dimensionality
• SGD Beats the Curse of Dimensionality
• Empirical Validation on Synthetic Data
• 5 LeJEPA: Stable and Scalable Implementation
• 5.1 LeJEPA: SIGReg + Prediction Loss
• 5.2 Relation to Prior Work
• 6 LeJEPA: Empirical Validation
• 6.1 LeJEPA’s Stability Across Hyper-Parameters and Architectures
• 6.2 LeJEPA’s Training Loss is Informative of Downstream Performance
• 6.3 In-Domain LeJEPA Outperforms Frontier Model Transfer Learning
• 6.4 LeJEPA Scales Across Data and Models
• 6.5 Emergent Semantic Structure in LeJEPA Representations
• 7 Conclusion
• Acknowledgments
• Appendix A Additional Details on Nonlinear Probing
• A.1 kNN Probing
• A.2 Kernel Probing
• Appendix B Proofs
• B.1 Proof of Section 3.1
• Proof.
• B.2 Proof of Section 3.1
• Proof.
• B.3 Proof of Section A.1
• Proof.
• B.4 Proof of Section A.1
• Proof.
• 1) The score-gradient term 𝔼 ⁡ [ A ​ ( X ) 2 ] \mathbb{E}[A(X)^{2}] .
• 2) The cross term 2 ​ 𝔼 ​ [ A ⁡ ( X ) ​ C ​ ( X ) ] 2\mathbb{E}[A(X)C(X)] .
• 3) The curvature term 𝔼 ⁡ [ C ​ ( X ) 2 ] \mathbb{E}[C(X)^{2}] .
• Putting it together.
• Proof.
• Step 2: Optimizing over covariance shapes under scalar constraints
• (a) Trace constraint.
• (b) Determinant constraint.
• (c) Frobenius-norm constraint.
• (d) Spectral-radius constraint.
• Conclusion: Isotropic Gaussian is optimal
• Proof.
• B.5 Proof of Section A.2
• Proof.
• B.6 Proof of Equation 6 to Equation 7
• Proof.
• B.7 Proof of Section A.2
• Proof.
• B.8 Proof of Section 4.1
• Proof.
• B.9 Proof of Section 4.1
• Proof.
• B.10 Proof of Section 4.3
• Proof.
• B.11 Proof of Section 4.2.1
• B.12 Proof of Section 4.2.3
• Proof.
• B.13 Proof of Section 5.1
• Proof.
• B.14 Proof of VICReg’s Recovery
• Proof.
• Part I: 𝔼 ⁡ [ 𝐗 ] = 𝟎 \mathbb{E}[\mathbf{X}]=\mathbf{0}
• Part II: Cov ⁡ ( 𝐗 ) = 𝐈 d \mathrm{Cov}(\mathbf{X})=\mathbf{I}_{d}
• Appendix C Background
• Appendix D Details on Low-Discrepancy Sequences
• Appendix E Shapiro-Wilk Test
• Appendix F Multivariate Statistics
• Instructions for reporting errors