LeJEPA: Provable and Scalable Self-Supervised Learning Without the Heuristics — part 4/10

Cleaned-text code points [45369, 60506). Bibliography omitted here; source.md includes it.

The Prediction Loss. To standardize notations, we adopt the DINO ( Caron et al., 2021 ) setup of generating V g V_{g} global views and V l V_{l} local views, leading to a total of V = V g + V l V=V_{g}+V_{l} views. We set the first 1 , … , V g 1,\dots,V_{g} indices of each 𝒛 n , v {\bm{z}}_{n,v} as the global views. For the cases without local views, simply set V l = 0 V_{l}=0 . The prediction loss is then given by having all views predict the global views as ℒ pred ​ ( { 𝒛 n , v } v = 1 V ) = \displaystyle\mathcal{L}_{\rm pred}(\{{\bm{z}}_{n,v}\}_{v=1}^{V})= 1 V g ​ ∑ v = 1 V g 1 V ​ ∑ v ′ = 1 V ‖ 𝒛 n , v − 𝒛 n , v ′ ‖ 2 2 \displaystyle\frac{1}{V_{g}}\sum_{v=1}^{V_{g}}\frac{1}{V}\sum_{v^{\prime}=1}^{V}\left\|{\bm{z}}_{n,v}-{\bm{z}}_{n,v^{\prime}}\right\|_{2}^{2} (6) = \displaystyle= 1 V ​ ∑ v ′ = 1 V ‖ 1 V g ​ ∑ v = 1 V g 𝒛 n , v − 𝒛 n , v ′ ‖ 2 2 \displaystyle\frac{1}{V}\sum_{v^{\prime}=1}^{V}\left\|\frac{1}{V_{\rm g}}\sum_{v=1}^{V_{\rm g}}{\bm{z}}_{n,v}-{\bm{z}}_{n,v^{\prime}}\right\|_{2}^{2} (7) 1 V ​ ∑ v ′ = 1 V ‖ 𝝁 n − 𝒛 n , v ′ ‖ 2 2 , \displaystyle\frac{1}{V}\sum_{v^{\prime}=1}^{V}\left\|\bm{\mu}_{n}-{\bm{z}}_{n,v^{\prime}}\right\|_{2}^{2}, (8) where we denote 𝝁 n ​ \triangleq ​ 1 V g ​ ∑ v = 1 V g 𝒛 n , v \bm{\mu}_{n}\triangleq\frac{1}{V_{g}}\sum_{v=1}^{V_{g}}{\bm{z}}_{n,v} , the Equation 6 to Equation 7 derivations are detailed in Section B.6 .

LeJEPA Loss. The final total loss simply combines the above prediction loss along with SIGReg on each views as per ℒ LeJEPA ​ ( { 𝒙 n , v } n , v = 1 B , V ) = λ V ​ ∑ v = 1 V SIGReg ⁡ ( { { 𝒛 n , v } n = 1 B } ) + 1 − λ B ∑ n = 1 B ℒ ( V g ) pred ( { 𝒛 n , v } v = 1 V ) . \mathcal{L}_{\rm LeJEPA}(\{{\bm{x}}_{n,v}\}_{n,v=1}^{B,V})=\frac{\lambda}{V}\sum_{v=1}^{V}{\rm SIGReg}(\{\{{\bm{z}}_{n,v}\}_{n=1}^{B}\})\\ +\frac{1-\lambda}{B}\sum_{n=1}^{B}\mathcal{L}^{(V_{\rm g})}_{\rm pred}(\{{\bm{z}}_{n,v}\}_{v=1}^{V}). (9)

We present ( 9 )’s implementation in . Altogether, the entire implementation–besides the usual model definitions, optimizers, and data loaders–only takes a few dozens lines in PyTorch ( and ). The absence of prototypes, stop-gradients, and teacher-student networks makes ( 9 ) appealing as it only contains one hyperparameter, λ \lambda , balancing the trade-off between the prediction and isotropic Gaussian terms.

### 5.2 Relation to Prior Work

Prior to presenting our experiments ( Section 6 ), we conclude by discussing how our proposed LeJEPA and SIGReg objective relate to existing frameworks in the literature.

While there is no existing solution employing such slicing and distribution matching for JEPAs, there exists similar pipelines for generative models and optimal transport. Notably, the Sliced Score Matching ( Song et al., 2020 ) proposes to leverage univariate slicing of the space to ease the estimation of a density for generative models. In a similar vein, the sliced Wasserstein distance ( Bonneel et al., 2015 ; Nguyen and Ho, 2023 ) uses such strategy to speed up and improve optimal transport. Furthermore, when the integral of the ( Epps–Pulley ) test is computed exactly, as opposed to our quadrature, each slice loss value recovers the kernel MMD ( Sriperumbudur et al., 2010 ; Gretton et al., 2012 ; Chwialkowski et al., 2016 ) measuring the distance between two distributions–albeit with a quadratic complexity. Lastly, it is possible to recover some existing SSL frameworks in the limit by employing LeJEPA with a particular test–instead of the preferred ( Epps–Pulley ). For example, Setting T ⁡ ( { x n } n = 1 B ) = mean ​ ( { x n } n = 1 B ) 2 + ( std ⁡ ( { x n } n = 1 B ) − 1 ) 2 T(\{x_{n}\}_{n=1}^{B})={\rm mean}(\{x_{n}\}_{n=1}^{B})^{2}+({\rm std}(\{x_{n}\}_{n=1}^{B})-1)^{2} and using that T T with SIGReg in LeJEPA recovers the VICReg SSL method in the limit of large number of slices. In fact, SIGReg will enforce in expectation that 𝔼 ⁡ [ 𝐙 ] = 𝟎 \mathbb{E}[\mathbf{Z}]=\mathbf{0} and Cov ⁡ ( 𝐙 ) = 𝐈 d \mathrm{Cov}(\mathbf{Z})=\mathbf{I}_{d} , where 𝐈 d \mathbf{I}_{d} denotes the d × d d\times d identity matrix–derivations provided in Section B.14 . And since our invariance term is simply the ℓ 2 \ell_{2} distance between the views’ embeddings, LeJEPA recovers VICReg for this degenerate statistical test. Based on Section 4.2.1 , we however strongly advocate against such a setting as it would lead to shortcut solutions–a phenomenon already observed in VICReg.

## 6 LeJEPA: Empirical Validation

We now use the LeJEPA implementation described in Section 5.1 to demonstrate its effectiveness through comprehensive experiments. We show that LeJEPA: (i) trains reliably across diverse architectures and datasets ( Section 6.1 ), (ii) provides an informative training loss for model selection ( Section 6.2 ), (iii) outperforms frontier vision models on small-scale in-domain pretraining ( Section 6.3 ), (iv) scales successfully to nearly 1 billion parameters on ImageNet-1k ( Section 6.4 ), and (v) learns rich semantic segmentation features without explicit supervision.

### 6.1 LeJEPA’s Stability Across Hyper-Parameters and Architectures

We now demonstrate LeJEPA’s stability across hyperparameters, architectures, and experimental setups. Additional cross-domain stability results are presented in Section 6.3 .

(a) ( Epps–Pulley ) parameters

(b) Number of local/global views

(c) Mini-batch size

(d) Embedding/Projector dim.

(e) Register tokens

Stability across standard hyperparameters. We begin by evaluating LeJEPA on ImageNet-100 and ImageNet-1K. On ImageNet-100, we train a ResNet-50 and vary the number of views and the loss weighting λ \lambda ( Figure 8 ). Performance remains stable across both dimensions, leading us to recommend λ = 0.05 \lambda=0.05 as a robust default. On ImageNet-1K, we train a ViT-Large/14 and explore batch size, as well as the number of global ( V g V_{\rm g} ) and local ( V l V_{\rm l} ) views ( Table 1 b). We find that the configuration commonly used in prior work ( V g = 2 , V l = 8 V_{\rm g}=2,V_{\rm l}=8 ) transfers well to LeJEPA. Notably, LeJEPA achieves competitive performance with batch sizes as small as 128 on ImageNet-1K ( Table 1 c), suggesting reduced memory requirements compared to existing methods. We thus recommend to use λ = 0.05 \lambda=0.05 , V g = 2 V_{\rm g}=2 , V l = 8 V_{\rm l}=8 , and batch size ≥ 128 \geq 128 as starting points .

Stability across Epps-Pulley hyperparameters. We next examine hyperparameters specific to LeJEPA: the number of slices | 𝒜 | |\mathcal{A}| in SIGReg, the integration domain for the Epps-Pulley test ( Epps–Pulley ), and the number of quadrature points for numerical integration. Table 1 a shows ablations on ImageNet-1K with ViT-Large/14. Both the integration domain and number of quadrature points have negligible impact on performance. This is expected: since the characteristic function is accurate at zero, the moments of the distribution are well-characterized even with a modest integration range. The number of slices | 𝒜 | |\mathcal{A}| has a modest effect—while more slices slightly improve performance, even 512 slices yield competitive results. We thus recommend to use 17 integration points, an integration domain of [ − 5 , 5 ] [-5,5] , and 1024 slices as starting points .

Stability across architectures. A key advantage of LeJEPA over recent methods (e.g., IJEPA, DINOv2) is its architecture-agnostic design. While most modern self-supervised methods are tailored to Vision Transformers, LeJEPA works across diverse architecture families without modification. To validate this claim, we pretrain approximately 50 architectures from 8 different families on ImageNet-10, selecting all models in the timm library with fewer than 20M parameters. All models are able to learn high-quality representations reaching between 91.5% to 95% top 1 accuracy with frozen backbone linear probing. It seems that models performing well in supervised learning setups are also the ones to favor for LeJEPA, such as resnets and ViTs. We thus recommend to use standard architectures such as ResNets and ViTs over specialized models like EfficientNet as stating point.

Removal of popular heuristics. In addition to providing reliable performance across models and datasets, LeJEPA’s provable construction enables us to remove many heuristics traditionally used to prevent collapse. First, prior work has shown both empirically and theoretically that predictors in image JEPA (without asymmetric information) and teacher-student architectures serve primarily to prevent collapse ( Grill et al., 2020 ; Jing et al., 2021 ; Tian et al., 2021 ; Caron et al., 2021 ; Chen et al., 2021 ) . Removing these components produces collapsed encoders, i.e., with performances at chance-level. Thanks to LeJEPA’s SIGReg loss, we can remove both the predictor and teacher-student architecture without suffering from collapse, as shown in Table 4 . While a teacher-student configuration does provide a small performance boost for ViT models—consistent with observations in supervised learning via Stochastic Weight Averaging ( Izmailov et al., 2019 ) —it is not necessary to prevent collapse. In our setup, we apply SWA on the encoder producing μ \mu in Equation 7 . Second, recent work demonstrated that register tokens are needed to prevent training instabilities in vision models ( Oquab et al., 2023 ; Siméoni et al., 2025 ; Darcet et al., 2023 ) . We show in Table 1 that such instabilities likely stem from poorly conditioned training objectives. In contrast, LeJEPA does not require register tokens and achieves stable performance with or without them. We thus recommend training without a predictor or register tokens, and optionally applying SWA with ViTs for a possible performance gain.

### 6.2 LeJEPA’s Training Loss is Informative of Downstream Performance

A major challenge in SSL pretraining is the lack of reliable signals conveying the quality of the learned representation. As a result, it is common to monitor a supervised downstream task performance, sometimes supplemented with unsupervised embedding statistics ( Agrawal et al., 2022 ; Garrido et al., 2023 ; Thilak et al., 2023 ) . This process is highly limiting since it requires labeled data that is costly and overly specialized. This is further exacerbated in the latest JEPA models where training losses exhibit low correlation with downstream performance–and may not even decrease monotonically during training.

In contrast, we find that LeJEPA’s training loss behaves much more favorably–providing us with a meaningful signal on model quality. First, we provide in Figure 10 , the 2D plane spanned by the SIGReg and prediction losses where a clear trend with downstream task accuracy can be observed. More strikingly, the combined training loss ( 9 ) with mixing coefficient λ \lambda exhibits very high Spearman correlation ( Spearman, 1961 ) , denoted as ρ s \rho_{s} , of about 85 % 85\% with downstream accuracy–which is considered a strong signal. This strong relationship holds across datasets and architectures. As a result, a lower LeJEPA training loss reliably indicates a better downstream performance.

We can further improve this correlation through a simple scaling law based upon the trade-off weighting hyperparameter λ \lambda C ( α ) = ρ s ​ ( train_loss λ α , test_accuracy ) . \displaystyle C^{(\alpha)}=\rho_{s}\left(\frac{\text{train\_loss}}{\lambda^{\alpha}},\text{test\_accuracy}\right). (10) By setting α ≈ 0.4 \alpha\approx 0.4 , LeJEPA’s training loss is able to achieve nearly 99% correlation with downstream performance across multiple datasets and models. We depict the changes in C ( α ) C^{(\alpha)} as a function of α \alpha on multiple datasets and models in Figure 11 , as well as the training LeJEPA loss against downstream performance in Figure 19 . The strong alignment between LeJEPA’s training loss and model quality enables label-free SSL model selection and cross-validation .

### 6.3 In-Domain LeJEPA Outperforms Frontier Model Transfer Learning

A key promise of self-supervised learning is to learn universal representations that generalize across tasks and domains. However, current frontier foundation models (e.g., DINOv2/v3, IJEPA) are pretrained on natural images forcing practitioners in specialized domains to collect large amount of labels for supervised finetuning. In fact, most frontier models can not be trained directly on those domains as the number of samples may be small and searching again for the hyper-parameters would be cumbersome yet necessary ( Assran et al., 2022 ) .

To demonstrate LeJEPA’s versatility and ability to resolve that current pain-point, we propose to pretrain directly on a new domain without any change in the loss or the pretraining pipeline. We select the Galaxy10 dataset, a galaxy morphology classification task that differs significantly from natural images in both visual structure and statistical properties ( Balestriero et al., 2025 ) . The dataset contains 11,000 training samples across 10 galaxy types. For LeJEPA, we use the default hyper-parameters and pretrain for 400 epochs a variety of backbones. We compare against the latest DINOv2, DINOv3 and IJEPA. We report in Figure 12 the top1 accuracy for linear probing both with frozen backbone and full-finetuning. We observe that in-domain pretraining with LeJEPA substantially outperforms state-of-the-art frontier models (DINOv2, DINOv3) on both linear probing and full finetuning . Additional datasets and backbones are provided in Table 5 depicting LeJEPA’s ability to train in-domain, even with a dataset with 1000 1000 samples (flowers102). Coupling this result with the stability of LeJEPA across architectures and hyper-parameters should offer a promising alternatives in domains not yet accounted for by the latest frontier models.

### 6.4 LeJEPA Scales Across Data and Models

We now propose to apply LeJEPA over a larger pretraining dataset, i.e., Imagenet-1k, and over larger backbones such as ViT/Large (0.3B), ConvNextV2-Huge (0.6B). For those two models, we reach an online linear probe accuracy on inet1k of 77.1% and 78.5% respectively. Beyond in-distribution performances, we also explore transfer learning. For those experiments, our baselines are IJEPA with a ViT-Huge (0.6B) which is the closest to our setup, and we also include a recent improved version of IJEPA including additional stochastic prediction tasks ( Bar et al., 2023 ) that is coined IJEPA + STOP. For LeJEPA, we employ the same recipe as described in Section 6.1 and report transfer learning performances with frozen backbone in Table 2 . We observe that we consistently outperform IJEPA while employed a smaller model and shorted training schedule. Beyond top1 accuracy, we also echo our findings from Section 6.2 about LeJEPA’s training loss quality. In our setup, we observe a very stable and smooth training curve indicating a stable optimization landscape removing the need for careful hyperparameter selection (recall Section 4.2.3 ). We provide an example on a ViT-gigantic (1.8B parameters) in id1 .

### 6.5 Emergent Semantic Structure in LeJEPA Representations