LeJEPA: Provable and Scalable Self-Supervised Learning Without the Heuristics — part 1/10

Cleaned-text code points [0, 15344). Bibliography omitted here; source.md includes it.

LeJEPA: Provable and Scalable Self-Supervised Learning Without the Heuristics Randall Balestriero 1,2 ,* Yann LeCun 3,2 ,* 1 Brown University 3 New York University (NYU) 2 Meta-FAIR * Equal contribution

Learning manipulable representations of the world and its dynamics is central to AI. Joint-Embedding Predictive Architectures (JEPAs) offer a promising blueprint, but lack of practical guidance and theory has led to ad - hoc R&D. We present a comprehensive theory of JEPAs and instantiate it in LeJEPA , a lean, scalable, and theoretically grounded training objective. First, we identify the isotropic Gaussian as the optimal distribution that JEPAs’ embeddings should follow to minimize downstream prediction risk. Second, we introduce a novel objective– Sketched Isotropic Gaussian Regularization (SIGReg)–to constrain embeddings to reach that ideal distribution. Combining the JEPA predictive loss with SIGReg yields LeJEPA with numerous theoretical and practical benefits: (i) single trade - off hyperparameter, (ii) linear time and memory complexity, (iii) stability across hyper-parameters, architectures (ResNets, ViTs, ConvNets) and domains, (iv) heuristics-free, e.g., no stop - gradient, no teacher–student, no hyper-parameter schedulers, and (v) distributed training-friendly implementation requiring only ≈ \approx 50 lines of code. Our empirical validation covers 10+ datasets, 60+ architectures, all with varying scales and domains. As an example, using imagenet-1k for pretraining and linear evaluation with frozen backbone, LeJEPA reaches 79% with a ViT-H/14. We hope that the simplicity and theory-friendly ecosystem offered by LeJEPA will reestablish self-supervised pre-training as a core pillar of AI research ( GitHub repo ). Full FT Frozen Method 1-sh Full 1-sh Full LeJEPA (in-domain) ConvNeXt-V2 Nano 29.42 82.72 28.74 76.52 ResNet-34 24.27 83.28 31.08 78.17 Frontier (transfer) DINOv2 ViT-S/16 21.05 78.34 27.68 67.62 DINOv3 ViT-S/16 24.71 81.60 30.17 71.38 Figure 1: LeJEPA overview. Top-left: Training loss exhibits strong correlation with downstream linear probe performance on ImageNet-1k (ViT-base), providing the first practical loss for model selection without supervised probing. Top-right: Training stability without heuristics even on 1.8B ViT-g models, stable training loss. Bottom-left: PCA features from ImageNet-1k pretrained LeJEPA ViT-Large demonstrate clear semantic relationships. Bottom-right: Galaxy10 in-domain results showcasing LeJEPA’s in-domain pretraining consistently outperforms state-of-the-art frontier foundation models transfer learning (DINOv2/v3 trained on natural images) across data regimes from 1-shot to full supervision. This demonstrates that domain-specific SSL beats generic transfer learning , even against massive-scale frontier models, when the framework scales effortlessly to any domain, model, and data scale.

f 𝜽 → \underset{\rightarrow}{f_{{\bm{\theta}}}}

## 1 Introduction

Learning manipulable representations of the world and its dynamics is a long - standing question in AI, with roots dating back centuries ago ( Von Helmholtz, 1867 ; Tolman, 1948 ; Gregory, 1980 ; Sutton, 1991 ; Friston, 2010 ) . Across domains, e.g., image recognition, robotics, physics, space exploration, the unifying question is how to learn an organized and actionable high - dimensional embedding space from observations? Using Deep Networks–parameterized nonlinear operators f 𝜽 f_{{\bm{\theta}}} –to map observations to embeddings is a standard first piece of that puzzle ( LeCun et al., 2015 ; Goodfellow et al., 2016 ) . The second, less standardized, piece of that puzzle is how to train f 𝛉 f_{{\bm{\theta}}} . Joint-Embedding Predictive Architectures (JEPAs) suggest training f 𝜽 f_{{\bm{\theta}}} by maximizing predictive agreement between the embeddings of semantically related views ( Bromley et al., 1993 ; LeCun, 2022 ; Balestriero et al., 2023 ) . Views can come in two forms: transformations or corruptions. They can involve masking, cropping, blurring, temporal or spatial translations, geometric or photometric transformations, viewpoint changes, views from different sensor modalities, etc. The supervised forms involve human-produced components such as image-caption pairs, text-code pairs, etc ( Tian et al., 2020 ) . In any case, views are expected to share some degree of semantic relationship to allow the prediction task to align f 𝜽 f_{{\bm{\theta}}} ’s embeddings towards the underlying knowledge present in the data.

Alas, JEPA’s prediction task admits failure modes, such as representation collapse, where f 𝜽 f_{{\bm{\theta}}} maps all inputs to nearly identical embeddings ( complete collapse ) or to a low-dimensional subspace ( dimensional collapse ) ( Jing et al., 2021 ) ( Jing et al., 2021 ; Cosentino et al., 2022 ; Balestriero and LeCun, 2022 ) . To mitigate such shortcut solutions, state - of - the - art recipes rely on heuristics–stop - gradient ( Chen et al., 2020a ) , asymmetric view generation ( Wang et al., 2022 ) , teacher–student networks with carefully tuned EMA schedules ( Caron et al., 2021 ; Tian et al., 2021 ) , explicit normalization and whitening layers ( Ermolov et al., 2021 ; Chen et al., 2021 ) –and a delicate balance of hyperparameters. As a result, today’s JEPA training is brittle and most research has shifted toward scaling data ( Vo et al., 2024 ) , models ( Fan et al., 2025 ) and even post-training Rodas et al. (2025) while leaving the theoretical foundations of JEPAs largely unexplored.

Our study proposes to break that cycle by questioning some of the fundamental design principles underpinning JEPAs. That introspection will start by asking what are the necessary conditions that JEPAs should abide by? Those minimal conditions will then act as axioms for us to design a novel and lean JEPA. We identify two axioms: (i) solving the prediction task while (ii) enforcing an isotropic Gaussian distribution of the embeddings ( Section 3 ). While (i) follows standard practice ( Balestriero and LeCun, 2022 ) , we introduce in Section 4 a novel distribution matching objective–Sketched Isotropic Gaussian Regularization (SIGReg)–to enforce (ii). The use of SIGReg not only removes the need for the numerous heuristics previously employed to prevent representation collapse, but SIGReg also exhibits favorable scaling properties as its memory and computational complexity is linear in dimension and sample size . Crucially, SIGReg’s isotropic Gaussian enforcement solves the collapsed shortcut solution and provably minimizes the model’s expected risk over the space of downstream tasks to be encountered post-training. The resulting JEPA solution–coined Latent-Euclidean JEPA (LeJEPA)–is introduced in Section 5 . Beyond theoretical optimality, LeJEPA offers numerous benefits such as (i) provable statistical guarantees, (ii) removal of heuristics such as teacher-student networks, (iii) linear memory and computational complexity, and most importantly (iv) a unified design with a single trade-off parameter that works out of the box across datasets, architectures and scales (see Section 6 ). We summarize our contributions below.

Contribution 1: We prove the optimal embedding distribution for foundation models. We establish that the isotropic Gaussian uniquely minimizes downstream prediction risk across broad task families. In Section 3 , we derive this result rigorously for both linear ( Section 3.1 ) and nonlinear probes ( Section 3.2 ), providing the first principled answer to what distribution f 𝜽 f_{{\bm{\theta}}} ’s embeddings should follow. This theoretical result transforms JEPA design from heuristic exploration to targeted optimization. Contribution 2: We introduce SIGReg, a distribution matching objective that uniquely combines provable correctness with computational efficiency at scale. We present Sketched Isotropic Gaussian Regularization (SIGReg), a novel objective that enforces distributional alignment via random projections and characteristic-function matching ( Sections 4 and 2 ). SIGReg provides statistical guarantees ( Sections 4.1 and 4.2 ) while achieving linear complexity and bounded gradients—a combination that existing distribution matching methods do not offer. Critically, its projection-based construction defeats the curse of dimensionality ( Section 4.3 ), making it both theoretically sound and practically efficient for high-dimensional embeddings.

Contribution 3: We design LeJEPA, a statistically optimal JEPA that eliminates collapse by construction. By combining JEPA’s predictive objective with SIGReg targeting the isotropic Gaussian, we introduce LeJEPA —Latent-Euclidean JEPA ( Section 5 ). LeJEPA requires only a single hyperparameter, eliminates representational collapse without stop-gradients or teacher-student architectures, and transfers across architectures and datasets without hyperparameter tuning. This demonstrates that principled theory directly yields practical simplicity.

Contribution 4: We validate LeJEPA at scale across diverse architectures and establish in-domain pretraining as viable. Our experiments ( Section 6 ) span ViTs, ConvNeXts, ResNets, MaxViTs, and Swin Transformers at scales approaching 1 billion parameters, where LeJEPA matches or exceeds state-of-the-art methods while maintaining training simplicity and robustness. Critically, on domain-specific datasets (Galaxy10, Food101), LeJEPA outperforms DINOv2-based transfer learning when pretrained directly on target data. This challenges the transfer learning paradigm and demonstrates that principled SSL can unlock effective in-domain pretraining—previously considered impractical for small datasets.

## 2 Background and Notations

We start by introducing some of the notations we will be using throughout our manuscript ( Section 2.1 ), followed by a review of JEPAs ( Section 2.2 ), and existing literature studying their design ( Section 2.3 ).

### 2.1 Notations and Definitions

Data. We are in possession of a dataset of shape ( N , V , D ) ∈ ℕ ∗ 3 (N,V,D)\in{\mathbb{N}^{*}}^{3} where N N is the number of samples, V V is the number of views, and D D is the dimension. One entry of this dataset is accessed via 𝒙 n , v , d {\bm{x}}_{n,v,d} . Those dimensions are often interpreted as follows: ( N ) is the number of independent samples, e.g., different images or different videos, ( V ) is the number of views , e.g., data-augmentations for images, frames for videos, and ( D ) is the dimension of each 𝒙 n , v {\bm{x}}_{n,v} , e.g., number of RGB pixels for images. In many cases the ordering over V V is given by time –but in some cases, e.g., data-augmentation of an image, ordering becomes irrelevant. Our study does not require any particular choice to organize one’s dataset into a ( N , V , D ) (N,V,D) tensor– and none of our theory and implementation assumes a particular design decision for that tensor . However, we will rely on the following two properties, ( independence ) the samples 𝒙 n , 𝒙 n ′ {\bm{x}}_{n},{\bm{x}}_{n^{\prime}} have been obtained independently from each other ∀ n ≠ n ′ \forall n\not=n^{\prime} , and ( identically distributed ) the sampling process was identical among 𝒙 n , ∀ n {\bm{x}}_{n},\forall n .

Deep Networks. Today’s AI solutions rely on Deep (Neural) Networks (DNs), which are compositions of a large number of parameterized linear and nonlinear operators. We denote the DN’s mapping as f 𝜽 : ℝ D → ℝ K f_{\bm{\theta}}:\mathbb{R}^{D}\rightarrow\mathbb{R}^{K} with K K the dimension of the embedding space. The internals of f 𝜽 f_{\bm{\theta}} are designed by the researcher to incorporate as much prior knowledge about the data as possible. The details of f 𝜽 f_{\bm{\theta}} are irrelevant to our study–as we will see the proposed LeJEPA works out-of-the-box on any f 𝜽 f_{\bm{\theta}} . In any case, all the learnable parameters are gathered in the vector 𝜽 ∈ ℝ P {\bm{\theta}}\in\mathbb{R}^{P} , with P P counting the total number of parameters. A central challenge in AI research is to design the right architecture and training objective so that 𝜽 {\bm{\theta}} can be learned from gradient descent to ultimately produce a useful system, or foundation model, f 𝜽 f_{{\bm{\theta}}} .

JEPAs. A foundation model is any system, e.g., a DN, able to solve numerous downstream tasks without requiring any change in its internal parameters 𝜽 {\bm{\theta}} . This is in sharp contrast with a supervised model that only considers its training task. JEPAs have formally been introduced by LeCun (2022) as a vehicle to produce foundation models. The core building blocks of JEPAs rely on numerous well-established techniques such as siamese networks ( Bromley et al., 1993 ) and predictive coding ( Helmholtz and others, 1867 ; Bruner and Postman, 1949 ) . While the exact blueprint of JEPAs varies greatly between use-cases, they all rely on two core principles: (i) being able to predict the embedding of a view 𝒙 n , v {\bm{x}}_{n,v} from the embedding of another view 𝒙 n , v ′ , v ′ ≠ v {\bm{x}}_{n,v^{\prime}},v^{\prime}\not=v , all while (ii) ensuring that the embeddings do not become degenerate. Concretely, once a JEPA is designed and trained, it should be able to solve numerous downstream tasks in zero or few shots. The JEPA objective function, along with some examples for 𝒙 {\bm{x}} , is provided in Equation 1 . The predictability criterion can be done by directly comparing the embeddings of the partial views E ​ n ​ c ​ ( 𝒙 n , v , . ) Enc({\bm{x}}_{n,v,.}) and E ​ n ​ c ​ ( 𝒙 n , v ′ , . ) Enc({\bm{x}}_{n,v^{\prime},.}) with a metric, e.g., ℓ p \ell_{p} . In some cases, an additional DN coined Pred , is employed to compare P ​ r ​ e ​ d ​ ( E ​ n ​ c ​ ( 𝒙 n , v , . ) ) Pred(Enc({\bm{x}}_{n,v,.})) against E ​ n ​ c ​ ( 𝒙 n , v ′ , . ) Enc({\bm{x}}_{n,v^{\prime},.}) –which is only justified when there exists an asymmetry between the information content of the different views, e.g., by conditioning the predictions on observed actions from robotics data ( Khazatsky et al., 2024 ) .

### 2.2 The Need for Reliable Pretraining

The JEPA’s prediction task is designed based on a priori knowledge of the data. Its design is often quite natural since it is relatively intuitive to form 𝒙 {\bm{x}} so that its views share the relevant information content one hope to capture. On the other hand, the design of the “anti-collapse” criterion is much closer to a game of Whac-A-Mole. Today’s designs rely on many different under-specified safeguards which are carefully combined in the hope that degenerate shortcut solutions are avoided during training. Such mechanisms include (i) feature whitening ( Ermolov et al., 2021 ; Bardes et al., 2021 ) , (ii) negative samples ( Chen et al., 2020a ; He et al., 2020 ) , and (iii) asymmetric views and teacher-student networks with stop-gradient ( Caron et al., 2021 ; Assran et al., 2023 ) . Those mechanisms all suffer from at least two of the following limitations: (i) under-specification, i.e., the criteria can be minimized while embeddings are in a degenerate configuration, (ii) quadratic time and memory complexity with mini-batch size and/or embedding dimension, (iii) sensitivity to data distribution, hyperparameters, architecture, and (iv) lack of theoretical understanding and guarantees.

### 2.3 The Need for Actionable Theory