VISReg: Variance-Invariance-Sketching Regularization for JEPA training — part 3/4

Cleaned-text code points [31011, 46784). Bibliography omitted here; source.md includes it.

VISReg has a better OOD performance. Due to the lack of OOD evaluations in previous work, we select 6 datasets from distinct domains: ChestXRay, RetinaMNIST, and OrganAMNIST are from medical domain, Galaxy10 is from space domain, and AID has the aerial images. The results in Table 6 suggest that VISReg helps the model learn more general features than the other methods. Without using training heuristics, VISReg achieves the best average accuracy comparing with all methods and backbone scales. Moreover, after scaling the training set to ImageNet22K, VISReg with ViT-L/14 backbone achieves a comparable accuracy to DINOv2, which was trained with a 10x larger training set. This indicates the generality of the representations learned by VISReg. This advantage also benefits the transfer learning capability.

VISReg has a good transfer learning capability. We conduct a transfer learning experiment on CIFAR10 & CIFAR100, Flowers, ImageNet1K, and Galaxy10. To have a fair comparison with DINO, the backbone is ViT-B/16 and the fine-tuning follows DINO ( Caron et al., 2021 ) implementation. An important observation is that, although VISReg does not have a better linear projection accuracy on in-domain datasets than DINO, the fine-tuning results are consistently higher than both supervised learning ( Touvron et al., 2021 ) and DINO. In addition, the advantage of VISReg on OOD datasets still remains.

VISReg shows an on-par performance on dense instance prediction. Following DINOv2 ( Oquab et al., 2024 ) , we conduct a simple linear segmentation experiment on ADE20K and the mIoU result is reported. Table 8 indicates that, without using any heuristics, VISReg can still provide a good segmentation results. Nevertheless, there is still a large gap comparing with MoCoV3 and iBOT, which is an important aspect that we will work on.

VISReg provides a good guidance to speed up the training of generative models. Another important application of foundation model is to speed up the training process of generative model ( Yu et al., 2025 ; Singh et al., 2025 ) . We use the official code of iREPA ( Singh et al., 2025 ) and run a lightweight training on SiT-B/2 for 100K steps with the features from VISReg and DINO. We use the default settings for both training and generation. The results in Table 9 suggest that VISReg provides useful embeddings.

## 5 Conclusion

This paper proposes VISReg, a self-supervised learning method that does not rely on heuristics for training stability. We present its effectiveness on model scaling, training stability, and training efficiency. In addition, we show that VISReg and its alike method is more robust to low-quality datasets than DINO, which is helpful in real-world applications. Last, we conduct extensive experiments to evaluate its performance in important aspects of a foundation model training method. It is intriguing that VISReg has a stronger performance on OOD data and transfer learning. With this potential, we hope this technical path can enhance the usefulness of the foundation models.

## 6 Acknowledgment

We appreciate Prof. Yann LeCun’s efforts in connecting resources and people to bring this project to fruition.

## Appendix A Implementation details

### A.1 Training details

Pretraining on ImageNet-1K. We pretrain two model variants on ImageNet-1K ( Deng et al., 2009 ) (1.28M training images): VISReg-B (ViT-B/16, 86M parameters) and VISReg-L (ViT-L/14, 304M parameters). Both models are trained from scratch using the VISReg regularization objective and timm for backbones.

We adopt DINO-style multi-crop augmentation ( Caron et al., 2021 ) : each image produces N g = 4 N_{g}{=}4 global crops ( 224 × 224 224{\times}224 , scale [ 0.3 , 1.0 ] [0.3,1.0] ) and N l = 6 N_{l}{=}6 local crops ( 96 × 96 96{\times}96 for ViT-B, 98 × 98 98{\times}98 for ViT-L, scale [ 0.05 , 0.3 ] [0.05,0.3] ), yielding 10 views per image. Augmentations include random horizontal flip, color jitter ( p = 0.8 p{=}0.8 ), random grayscale ( p = 0.2 p{=}0.2 ), Gaussian blur ( p = 0.5 p{=}0.5 ), and random solarize ( p = 0.2 p{=}0.2 ).

We use AdamW with weight decay 5 × 10 − 2 5{\times}10^{-2} and bfloat16 mixed precision. The learning rate follows a linear warmup over 5 epochs then cosine annealing to lr max / 1000 \mathrm{lr}_{\max}/1000 . Projections are produced by a 3-layer MLP ( 2048 → 2048 → d p 2048\to 2048\to d_{p} ) with batch normalization and GELU activations, applied to the concatenated CLS tokens from the last two backbone layers.

VISReg-B uses learning rate 9 × 10 − 4 9{\times}10^{-4} , λ = 0.9 \lambda{=}0.9 , projection dimension d p = 256 d_{p}{=}256 , K = 2048 K{=}2048 random projections for VISReg, and per-GPU batch size 16 (effective batch size 512 across 32 GPUs). VISReg-L uses learning rate 8 × 10 − 4 8{\times}10^{-4} , λ = 0.7 \lambda{=}0.7 , d p = 384 d_{p}{=}384 , K = 4096 K{=}4096 random projections, and per-GPU batch size 16 (effective batch size 512 across 32 GPUs). Both models are trained for 400 epochs on 32 NVIDIA H100 80GB GPUs (4 nodes × \times 8 GPUs) using HuggingFace Accelerate for distributed training, requiring approximately 1,120 and 2,060 GPU-hours for ViT-B and ViT-L, respectively.

Pretraining on ImageNet-22K. We additionally pretrain VISReg on ImageNet-22K ( Deng et al., 2009 ) (14.2M images) with ViT-L/14 for 100 epochs on 16 NVIDIA H100 80GB GPUs (4 nodes × \times 4 GPUs). The multi-crop strategy uses N g = 2 N_{g}{=}2 global crops and N l = 8 N_{l}{=}8 local crops ( 98 × 98 98{\times}98 ), still yielding 10 views per image. We use per-GPU batch size 64 (effective batch size 1,024), learning rate 8 × 10 − 4 8{\times}10^{-4} , λ = 0.8 \lambda{=}0.8 , d p = 384 d_{p}{=}384 , and K = 4096 K{=}4096 random projections. All other settings (optimizer, scheduler, projector architecture) follow the ImageNet-1K configuration. Training requires approximately 2,720 GPU-hours.

### A.2 Testing details

Downstream classification. We evaluate pretrained representations on 8 classification benchmarks following the DINOv2 linear evaluation protocol ( Oquab et al., 2024 ) : DTD ( Cimpoi et al., 2014 ) , FGVC-Aircraft ( Maji et al., 2013 ) , Stanford Cars ( Krause et al., 2013 ) , CIFAR-10, CIFAR-100 ( Krizhevsky et al., 2009 ) , Oxford Flowers-102 ( Nilsback & Zisserman, 2008 ) , Food-101 ( Bossard et al., 2014 ) , and Oxford-IIIT Pets ( Parkhi et al., 2012 ) . We additionally evaluate on 6 out-of-distribution benchmarks: DTD, Galaxy10 ( Leung, 2025 ) , AID ( Xia et al., 2017 ) , NIH ChestX-ray ( Wang et al., 2017 ) , RetinaMNIST, and OrganAMNIST ( Bilic et al., 2023 ) . The details of each dataset can be found under A.3 .

For all classification tasks, we freeze the pretrained encoder and extract features by concatenating the CLS tokens from the last 4 transformer layers, yielding a feature vector of dimension 4 × d embed 4\times d_{\mathrm{embed}} (3,072 for ViT-B, 4,096 for ViT-L). A linear classifier with SyncBatchNorm is trained on top of these frozen features using SGD with momentum 0.9, no weight decay, and cosine annealing for 10 epochs with batch size 32. We perform a grid search over 13 base learning rates scaled by the linear scaling rule (effective batch size / 256), and report the best test accuracy. All images are resized to 224 × 224 224{\times}224 with standard ImageNet normalization.

ImageNet-1K linear probe. For ImageNet-1K linear evaluation, we follow the same frozen-feature protocol but with a dedicated multi-head implementation for efficiency. A multi-head linear classifier with shared SyncBatchNorm trains 10 independent heads in parallel (one per learning rate), for 100 epochs using SGD with momentum 0.9, no weight decay, and per-step cosine annealing. Training uses bfloat16 mixed precision on 8 GPUs with per-GPU batch size 32 (effective batch size 256). Standard ImageNet evaluation preprocessing is applied: random resized crop to 224 × 224 224{\times}224 for training, resize to 256 then center crop to 224 for validation. The best accuracy across all heads on the 50K validation set is reported.

Semantic segmentation. We evaluate on ADE20K ( Zhou et al., 2017 ) (150 classes) using a linear segmentation probe. A single 1 × 1 1{\times}1 convolution with SyncBatchNorm is trained on frozen patch features from the last transformer layer. Training uses AdamW with learning rate 2 × 10 − 3 2{\times}10^{-3} , polynomial LR decay (power 0.9), batch size 16, and image size 518 × 518 518{\times}518 (for patch-14 models) or 512 × 512 512{\times}512 (for patch-16 models) for 40 epochs. We report mean intersection-over-union (mIoU) on the validation set.

### A.3 Datasets

We evaluate our pre-trained models on a diverse set of 15 datasets spanning multiple domains, tasks, and difficulty levels. The following sections describe each dataset in detail.

ImageNet-1k ( Deng et al., 2009 ) is a large-scale object recognition dataset containing 1,281,167 training images and 50,000 validation images across 1,000 classes, representing diverse natural objects, animals, and scenes from the natural world.

CIFAR-10 ( Krizhevsky et al., 2009 ) is a 10-class object classification dataset with 50,000 training and 10,000 test images at 32 × 32 32\times 32 resolution, covering categories like airplanes, cars, birds, cats, and other common objects.

CIFAR-100 ( Krizhevsky et al., 2009 ) is a fine-grained object classification dataset with 50,000 training and 10,000 test images at 32 × 32 32\times 32 resolution, containing 100 classes organized into 20 supercategories including various vehicles, animals, and household items.

Stanford Cars ( Krause et al., 2013 ) is a fine-grained vehicle classification dataset with 8,144 training and 8,041 test images covering 196 car models from 98 manufacturers, spanning decades of automotive design from 1950 to 2012.

Galaxy10 ( Leung, 2025 ) is an astronomical image classification dataset with 17,736 images classifying galaxies into 10 morphological categories (disturbed, merging, spiral, elliptical, etc.) from the DECaLS survey.

Food-101 ( Bossard et al., 2014 ) is a food classification dataset with 75,750 training and 25,250 validation images covering 101 food categories including dishes like pizza, sushi, hamburger, and various international cuisines.

Oxford-IIIT Pets ( Parkhi et al., 2012 ) is a pet breed classification dataset with 3,680 training and 3,669 test images covering 37 cat and dog breeds, requiring fine-grained distinction between similar-looking breeds.

NIH Chest X-ray ( Wang et al., 2017 ) is a multi-label medical image classification dataset comprising 112,120 total images, where each chest radiograph may contain multiple pathology labels from 14 disease categories including pneumonia, cardiomegaly, and pleural effusion.

RetinaMNIST ( Liu et al., 2022 ) is a medical image classification dataset with 1,080 training, 120 validation, and 400 test retinal fundus images classifying diabetic retinopathy into 5 severity grades.

OrganAMNIST ( Bilic et al., 2023 ) is a medical image classification dataset with 34,581 training, 6,491 validation, and 17,778 test CT axial slices classifying 11 body organ types including liver, kidney, spleen, and heart.

Oxford Flowers 102 ( Nilsback & Zisserman, 2008 ) is a fine-grained plant classification dataset with 1,020 training, 1,020 validation, and 6,149 test images covering 102 flower species with 40-258 images per class.

Describable Textures (DTD) ( Cimpoi et al., 2014 ) is a texture classification dataset with 1,880 training, 1,880 validation, and 1,880 test images across 47 texture categories (e.g., braided, dotted, fibrous) following a 10-fold cross-validation protocol.

FGVC-Aircraft ( Maji et al., 2013 ) is a fine-grained aircraft classification dataset with 6,667 trainval and 3,333 test images covering 100 aircraft variants from the FGVC-Aircraft 2013b benchmark.

AID ( Xia et al., 2017 ) is a remote sensing scene classification dataset with 10,000 images across 30 aerial scene categories including airports, beaches, forests, and urban areas, using a 10%/90% train/test split for SSL evaluation.

ADE20K ( Zhou et al., 2017 ) is a semantic segmentation dataset with 20,210 training and 2,000 validation images, containing pixel-level annotations for 150 semantic classes including objects, parts, and materials across indoor and outdoor scenes.

## Appendix B Additional ablations

Our additional ablations focus on VISReg design, including the necessity of scale, shape, center loss, the necessity of applying gradient detachment between scale and shape loss, and the effect of the loss weight on each component. We include a long-tailed dataset (ImageNet-LT), a low-rank dataset (Galaxy10), and a normal dataset (Imagenette) to cover a wider range of application scenarios. All experiments use a ViT-S/8 backbone at 128 × 128 128{\times}128 resolution with 4 augmented views, learning rate 10 − 3 10^{-3} , per-GPU batch size 32 across 8 GPUs (effective batch 256), λ = 0.6 \lambda=0.6 , projection dimension 256, and K = 4096 K{=}4096 random projections. ImageNet-LT and Galaxy10 train for 400 epochs; Imagenette for 800.

### B.1 Effect of decoupled components in training.

First, we knocked out each contribution of scale, shape, and center in the training to understand the effectiveness of each part. Since the result on ImageNette has shown a clear pattern, ImageNet-LT and Galaxy are not included. Table 11 shows that both scale and shape loss significantly impact the learning process: 1) Without scale loss, there is a 71.02% decrease in accuracy; Without shape loss, there is a 58.4% decrease in accuracy. As for the center loss, the accuracy difference is 0.41% in the final accuracy, but importantly, it increases the convergence speed. Hence, all three components are necessary.

Second, we check the usefulness of applying detachment between scale loss and shape loss. The general observation from Table 11 is that, despite the minor improvement, detachment helps the model achieve a higher performance across all three datasets. Therefore, we choose to use it across all the experiments.

### B.2 Effect of decoupled components in different training set scenarios.

We ablate the ratio between the three DSSO loss components, i.e. , scale, shape, and center, while keeping the total weight constant ( λ scale + λ shape + λ center = 3 \lambda_{\text{scale}}+\lambda_{\text{shape}}+\lambda_{\text{center}}=3 ) so that λ \lambda alone controls the overall regularization magnitude.

The shape component is the most impactful of the three DSSO objectives. On ImageNet-LT and Galaxy10, shifting weight toward shape monotonically improves accuracy, with shape 4:1 outperforming the equal baseline by +3.2% and +1.3%, respectively. Conversely, emphasizing scale or center consistently degrades performance, with scale 4:1 producing the largest drops (-4.6% on ImageNet-LT, -2.7% on Galaxy10). This suggests that a higher shape regularization helps the learning on low-quality datasets. However, the result on Imagenette shows that the default setting is the best choice. Other imbalanced ratio across three factors largely reduces the learning effectiveness.

## Appendix C Visualizations

### C.1 VISReg loss indicates the performance

Strong correlation between loss and online probe accuracy is a important advantage of the theorem proposed by LeJEPA ( Balestriero & LeCun, 2025 ) . We calculate the Pearson correlation between loss and online probe accuracy of the ViT-L training on ImageNet1K, as shown in Figure 7 . The pronounced -0.996 correlation show the loss curve can be used to reflect the learning curve of the model.

### C.2 Further comparison with DINOv1 on image and video.

#### PCA Feature Visualization.