Self-Supervised Learning from Images with a Joint-Embedding Predictive Architecture — part 3/3

Cleaned-text code points [30562, 45454). Bibliography omitted here; source.md includes it.

Because our I-JEPA implementation uses Vision Transformer architectures without a [cls] token, we adapt the default VISSL evaluation recipe to utilize the average-pooled patch representation instead of the [cls] token. We therefore report the best linear evaluation number among the following representations: 1) the average-pooled patch representation of the last layer, 2) the concatenation of the last 4 4 layers of the average-pooled patch representations. We otherwise keep the linear-probing recipe identical.

#### ImageNet evaluations.

To evaluate the I-JEPA on ImageNet [ 60 ] , we adapt the VISSL recipe to use average pooled representations instead of the [cls] token. Following MAE [ 36 ] , we use the LARS [ 72 ] optimizer with a batch-size of 16384 16384 , and train the linear probe for 50 epochs. We use a learning rate with a step-wise decay, dividing it by a factor of 10 10 every 15 15 epochs, and sweep three different reference learning rates [ 0.01 , 0.05 , 0.001 ] [0.01,0.05,0.001] , and two weight decay values [ 0.0005 , 0.0 ] [0.0005,0.0] .

#### Low-shot evaluation.

To evaluate our model on the ImageNet-1% low-shot task, we adapt the fine-tuning protocol of MAE [ 36 ] .We fine-tune our ViT-L/H models for 50 epochs on ImageNet-1% with the AdamW optimizer and a cosine learning rate scheduler. We use a batch size of 512, a learning rate layer decay of 0.75 0.75 and 0.1 0.1 label smoothing. We use the default randaugment data-augmentations as in MAE. In contrast to the fine-tuning done with MAE, we do not use mixup, cutmix, random erasing or drop path. For the I-JEPA, we use a learning rate /weight decay of 3e -5 /5e -2 for the ViT-L/16, 3e -5 /4e -1 for the ViT-H/14 and 3e -5 /4e -1 for the ViT-H/16 448 . Similar fine-tuning strategy for low-shot learning has been explored by Semi-VIT in the context of semi-supervised learning [ 16 ] .

## Appendix B Broader Related Work

Self-supervised learning of visual representations with joint-embedding architectures is an active line of research [ 69 , 37 , 24 , 35 , 23 , 18 , 10 , 79 , 12 , 54 , 3 ] . These approaches train a pair of encoders to output similar embeddings for two or more views of the same image. To avoid pathological solution, many popular joint-embedding approaches use explicit regularization [ 20 , 18 , 10 , 5 ] or architectural constraints [ 35 , 24 ] . Collapse-prevention based on architectural constraints leverage specific network design choices to avoid collapse, for example, by stopping the gradient flow in one of the joint-embedding branches [ 20 ] , using a momentum encoder in one of the joint-embedding branches [ 35 ] , or using an asymmetric prediction head [ 35 , 20 , 8 ] . Recent work [ 62 ] attempts to theoretically understand (in certain simplified settings) how joint-embedding methods with architectural constraints avoid representation collapse without explicit regularization.

Typical regularization-based approaches to collapse prevention in joint-embedding architectures try to maximize the volume of space occupied by the representations. This is often motivated through the InfoMax [ 52 ] principle. Indeed, a longstanding conviction in unsupervised representation learning is that the resulting representations should be both maximally informative about the inputs, while also satisfying certain simplicity constraints [ 50 , 33 ] . The former objective is often referred to as the information-maximization principle (InfoMax), while the latter is sometimes referred to as the parsimony principle [ 52 ] . Such approaches to representation learning have been proposed for decades (e.g., [ 14 ] ), where, historically, simplicity constraints were enforced by encouraging the learned representations to be sparse, low-dimensional, or disentangled, i.e., the individual dimensions of the representation vector should be statistically independent [ 33 ] . Modern approaches enforce the simplicity constraints coupled with InfoMax regularization through self-supervised loss terms [ 40 , 64 , 6 , 44 , 41 , 55 ] . One example is the widespread view-invariance penalty [ 53 ] , often coupled with with independence [ 74 , 10 ] or low-dimensionality constraints, e.g., by projecting representations on the unit hypersphere [ 20 , 37 , 35 ] . However, despite its proliferation, there have also been many criticisms of the InfoMax principle, especially since it is does not discriminate between different types of information (e.g, noise and semantics) [ 2 ] . Indeed, the sets of features we wish the model to capture are not always those with the highest marginal entropy (maximal information content).

Orthogonal to the contributions of invariance-based pretraining, another line of work attempts to learn representations by artificially masking parts of the input and training a network to reconstruct the hidden content [ 67 ] . Autoregressive models, and denoising autoencoders in particular, predict clean visual inputs from noisy views [ 19 , 67 , 36 , 9 , 8 ] . Typically, the goal is to predict missing inputs at a pixel level [ 29 , 36 , 70 ] , or at a patch token-level, using a tokenizer [ 9 , 68 ] . While these works demonstrate impressive scalability, they usually learn features at a low-level of semantic abstraction compared to joint-embedding approaches [ 4 ] .

More recently, a set of approaches attempt to combine both joint-embedding architectures and reconstruction based approaches [ 30 ] , wherein they combine an invariance pretraining loss with a patch-level reconstruction loss, as in the iBOT method [ 79 ] . Since view-invariance based approaches are typically biased towards learning global image representations, thereby limiting their applicability to other computer vision tasks, the idea is that adding local loss terms can improve performance on other popular tasks in computer vision [ 26 , 32 , 11 ] . The framework of contrastive predictive coding [ 55 ] is also closely related to this line of work on local loss terms. In the context of images [ 39 ] , here the idea is to use a contrastive objective combined with a convolutional network to discriminate between overlapping image patch representations. Specifically, the goal is to encourage the representations of an image patch to be predictable of the image patches directly below it, while pushing away the representations of other patch views. In contrast to that work, the proposed I-JEPA method is non-contrastive and does not seek to discriminate between image patches. Rather, the goal is to predict the representations of various target blocks from a single context block. This is achieved with a Joint-Embedding Predictive Architecture, using a predictor network that is conditioned on positional embeddings corresponding to the location of the target block in the image. Qualitative experiments in Section 8 show that the predictor network in our architecture learns to correctly perform this local-to-local region feature mapping, and learns to correctly capture positional uncertainty in the image.

## Appendix C Additional Ablations

This section follows the same experimental protocol as Section 9 . We report the result of a linear probe with a frozen backbone, trained on the low-shot 1% ImageNet-1K benchmark.

#### Multiblock masking strategy.

We present an extended ablation of the multiblock masking strategy where we change the targets block scale (Table 8 ), the context scale (Table 9 ) and the number of target blocks (Table 10 ). We train a ViT-B/16 for 300 epochs using I-JEPA with various multi-block settings and compare performance on the 1% ImageNet-1K benchmark using a linear probe. In short, we find that it is important to predict several relatively large (semantic) target blocks, and to use a sufficiently informative (spatially distributed) context block.

#### Masking at the output of the target-encoder.

An important important design choice in I-JEPA is that the target blocks are obtained by masking the output of the target-encoder, not the input. Table 11 shows the effect of this design choice on the semantic level of the learned representations when pretraining a ViT-H/16 using I-JEPA for 300 epochs. In the case where masking is applied to the input, we forward-propagate through the target-encoder once for each target region. Masking the output of the target-encoder during pretraining results in more semantic prediction targets and improves linear probing performance.

#### Predictor depth.

We examine the impact of the predictor depth on the downstream low-shot performance in Table 12 . We pretrain a ViT-L/16 for 500 epochs using either a 6-layer predictor network or a 12-layer predictor network. The model pretrained using a deeper predictor shows a significant improvement in downstream low-shot performance compared to the model pretrained with a shallower predictor.

#### Weight decay.

In Table 13 , we evaluate the impact of weight-decay during pretraining. We explore two weight decay strategies: linearly increase the weight-decay from 0.04 0.04 to 0.4 0.4 or use a fix weight-decay of 0.05 0.05 . Using a smaller weight decay during pretraining improves the downstream performance on ImageNet-1% when fine-tuning. However, this also leads to a degradation of performance in linear evaluation. In the main paper, we use the first weight decay strategy as it improves the performances in linear evaluation downstream tasks.

#### Predictor width.

We explore the impact of the predictor width in Table 14 . We compare I-JEPA using a ViT-L encoder and a predictor with 386 386 channels to a similar model using a predictor with 1024 1024 channels. Note that the ViT-L encoder has 1024 1024 channels. Using a bottleneck in the predictor width improves the downstream performance on ImageNet 1%.

## Appendix D Finetuning on the full ImageNet

In this section, we report performance on I-JEPA when fine-tuning on the full ImageNet dataset. We focus on the ViT-H/16 448 as this architecture achieves state-of-art performance with MAE [ 36 ] .

We use a fine-tuning protocol similar to MAE. Specifically, we fine-tune our model for 50 50 epochs using AdamW and a cosine learning rate schedule. The base learning rate is set to 10 − 4 10^{-4} and the batch size to 528 528 . We train using mixup [ 76 ] set to 0.8 0.8 , cutmix [ 73 ] set to 1.0 1.0 , a drop path probability of 0.25 0.25 and a weight decay set to 0.04 0.04 . We also use a layer decay of 0.75 0.75 . Finally, we use the same rand-augment data-augmentations as MAE,

Table 15 reports the fine-tuning results. I-JEPA achieves 87.1 87.1 top-1 accuracy. Its performance is less than 1 % 1\% away from the best MAE model despite I-JEPA being trained for 5.3 5.3 times less epochs than MAE. This result demonstrates that I-JEPA is competitive when fine-tuning on the full ImageNet dataset.

## Appendix E RCDM Visualizations

To visualize the representations of a pretrained neural network in pixel space, we use the RCDM framework [ 13 ] . The RCDM framework trains a decoder network h ω h_{\omega} , comprising a generative diffusion model, to reconstruct an image 𝒙 {\bm{x}} from the representation vector of that image 𝒔 x {\bm{s}}_{x} and a noisy version of that image 𝒙 ^ ≔ 𝒙 + ϵ \hat{{\bm{x}}}\coloneqq{\bm{x}}+\epsilon , where ϵ \epsilon is an additive noise vector. Concretely, the decoder objective is to minimize the loss function ∥ h ω ​ ( 𝒙 ^ , 𝒔 x ) − ϵ ∥ \lVert h_{\omega}(\hat{{\bm{x}}},{\bm{s}}_{x})-\epsilon\rVert . We train each RCDM network for 300,000 iterations using the default hyperparameters [ 13 ] . After training the decoder, one can subsequently feed the representation vector of an unseen test image 𝒔 y {\bm{s}}_{y} into the decoder along with various random noise vectors to generate several pixel-level visualizations of the representation, thus providing insight into the features captured in the representations of the pretrained network. Qualities that are common across samples represent information that is contained in the representation. On the other hand, qualities that vary across samples represent information that is not contained in the representations

In Figure 6 , the visualizations are obtained by feeding the average-pooled output of the predictor, conditioned on a specific target region, into the decoder network, along with various random noise vectors. In Figures 7 and 8 , the visualizations are obtained by feeding the average-pooled output of the target-encoder into the decoder network, along with various random noise vectors.

### E.1 Encoder Visualization

In Figure 7 , we visualize the average-pooled I-JEPA representations at the output of our ViT-H/14 target-encoder. The first column contains the original image, while subsequent columns contain synthetic samples obtained by feeding the average-pooled representation of the image into the decoder along with various random noise vectors. Figure 7 suggests that the I-JEPA target-encoder is able to correctly capture the high-level information regarding objects and their poses, while discarding low-level image details and background information.

Figure 8 shows similar visualizations, but when using an MSN [ 4 ] pretrained ViT-L/7 target-encoder to compute the image representations. The MSN method trains a context- and target-encoder using a Joint-Embedding Architecture to enforce invariance of global image representations to various hand crafted data augmentations and missing patches. While the MSN pretrained network is able to capture high level semantic information about the image in the first column, it also exhibits higher variability in the generated samples, e.g., variability in the object pose, object scale, and number of instances. In short, the MSN pretrained discards much of the local structure in the image, which is in stark contrast to I-JEPA, which retains information about much of the local structure in the input image.

## Instructions for reporting errors

We are continuing to improve HTML versions of papers, and your feedback helps enhance accessibility and mobile support. To report errors in the HTML that will help us improve conversion and rendering, choose any of the methods listed below:

Click the "Report Issue" ( ) button, located in the page header.

Tip: You can select the relevant text first, to include it in your report.

Our team has already identified the following issues . We appreciate your time reviewing and reporting rendering errors we may not have found yet. Your efforts will help us improve the HTML versions for all readers, because disability should not be a barrier to accessing research. Thank you for your continued support in championing open access for all.

Have a free development cycle? Help support accessibility at arXiv! Our collaborators at LaTeXML maintain a list of packages that need conversion , and welcome developer contributions .