A Cookbook of Self-Supervised Learning — part 4/8

Cleaned-text code points [46113, 61969). Bibliography omitted here; source.md includes it.

To provide other insights, we pretrained methods on Places205 ( Zhou et al., 2014 ) and iNaturalist18 ( Horn et al., 2018 ) without changing the augmentations strategy but tuning heavily loss related coefficients. The goal is to see if the setups used on ImageNet transfer well to other datasets. Places205 has the advantage of not being object centric, and iNaturalist of having a power law distribution of classes as well as requiring a lot of fine-grained information. We report our results in table 1 . As we can see most methods are able to achieve similar performance when pretraining either on ImageNet or on the target dataset. This would suggest that the protocol developed on ImageNet can transfer decently, since we noticed that hyperparameters that were optimal on ImageNet also tended to be on different datasets. There is one visbile exception though, SimCLR and MSN perform poorly on iNaturalist18 when pretraining on it directly. While conclusions are impossible to draw precisely here, it would suggest that certain method exhibit more sensitivity on the pretraining dataset than other.

##### Weakly-curated training data

: A successful approach to leverage large uncurated datasets is to perform retrieval in them based on curated data. This means that the dataset will contain images similar to a curated or smaller source dataset such as ImageNet, while being much larger and more diverse. This strategy was used in DINOv2 ( Oquab et al., 2023 ) where LVD-142M was built using a wide variety of small and domain specific datasets. While this does not lead to big performance boosts in classification on ImageNet, it can lead to significant boosts in performance on other tasks such as image retrieval.

## 3 A Cook’s Guide to Successful SSL Training and Deployment

### 3.1 Role of Data-Augmentation

Many SSL methods, especially joint embedding methods derived from Chen et al. (2020b) , require a way to define positive views from a given image to learn invariances. The proxy used in these SSL methods is to leverage data augmentation to define these invariances. For example, by using different crops of a given images and positive view, the SSL model will be trained to produce a representation that is invariant to these different crops. When using a grayscale operation, or a colorjitter one as positive views, the representation will have to be invariant to the color information. Thus, the deep nature of what is learned by the SSL models is defined by the data augmentation pipeline. It is worth noting that perfect invariance is not achieved thanks to the projector ( Bordes et al., 2022a ) , which helps improve performance on tasks which are not entirely invariant. Chen et al. (2020b) study how much influence have specific data augmentations on SimCLR with respect to the performances over ImageNet. They show that simpler data augmentation such as noise aren’t beneficial on ImageNet classification downstream. Instead, cropping and multiple color jittering operations lead to competitive results with a supervised baseline. This key element of data-augmentation had also been largely used in the following SSL works ( Chen et al., 2020d ; Bardes et al., 2021 ; Zbontar et al., 2021 ) without significant changes. The only variant that is sometimes used is adding smaller crops in addition of bigger crops when learning in-variances. We discuss this use of big and smaller crops, called multi-crop, in the coming subsections.

However this specific combination of data augmentation was specifically designed to reach good performances on ImageNet. Bordes et al. (2023a) study the impact of different choice of data augmentation on different downstream tasks and found that even if the addition of ColorJitter seem beneficial for many classification task it might not always be the case on other downstream tasks. Similarly, Ericsson et al. (2021a) show that different augmentations lead to learning different type of invariances for which some of them are better on some downstream tasks than other. The authors suggest to merge representations learned with different augmentations to improve transferability across a wider range of downstream task. There is also an hidden cost when using a complex pipeline of data augmentation: the data preprocessing time which might slow down significantly the training. Thus, when the training budget matter, it might be preferable to just use random crop along a grayscale operation when training a SSL model. We discuss common approaches for speeding up the training pipeline in Section 3.8.1 . Ni et al. (2021b) further show that contrastive learners can benefit from very aggressive data augmentations such as large rotations when explicitly trained not to be invariant to them, as in meta-learning ( Ni et al., 2021a ) .

Another line of work attempts to remove the need for these handcrafted data augmentations. One approach is to use a reconstruction-based objectives such as MAE ( He et al., 2022 ) which uses a reconstruction loss in pixel space to avoid the need for defining precise invariances. Another approach is based on a joint-embedding where based on random parts of an images the goal is to predict the representations of the missing parts of the image in the representation space. An example of such method is I-JEPA ( Assran et al., 2023 ) or Data2Vec2.0 ( Baevski et al., 2022 ) which use a context part of an image to predict missing small parts of the image. Another line of work tries to retain style information about the augmentations to improve downstream performance on tasks requiring style information such as color by predicting style information ( Xiao et al., 2020 ; Dangovski et al., 2021 ; Gidaris et al., 2018 ; Scherr et al., 2022 ) . Encoding true equivariance to augmentations (which requires a mapping between embedding) is an active line of work with approaches such as EquiMod ( Dangovski et al., 2021 ) , SEN ( Park et al., 2022 ) , or ( Marchetti et al., 2022 ) which also aims at splitting the representations as class and pose. This idea of splitting representations as invariant and equivariant was also explored in SIE ( Garrido et al., 2023 ) and using Lie group formalism in Ibrahim et al. (2022) .

#### 3.1.1 Role of multi-crop

While works such as MoCo ( Meng et al., 2021 ) are focused on increasing the number or quality of negative pairs, another direction to improve performance is to increase the number of positives for a given image. Multi-crop, which was introduced with SwAV ( Caron et al., 2020 ) , tackles this problem by introducing smaller crops ( 96 × 96 96\times 96 ) on top of the usual two large ones ( 224 × 224 224\times 224 ). Instead of only comparing the two large crops together, or all pairs of crops, the two large crops are each compared to all other crops (big or small). As such, if we have 2 large crops and N N small crops, the invariance loss is computed 2 ​ ( N − 1 ) 2(N-1) times, increasing the positive-pair related signal. The use of smaller crops as well as not comparing all pairs of crops helps reduce the computational cost of these additional crops. While the number of additional crops can vary (10 in Mugs ( Zhou et al., 2022b ) compared to 6 in SwAV), it always lead to an icrease in training time and memory usage if used as is. To mitigate this cost, using 160 × 160 160\times 160 large crops and 4 96 × 96 96\times 96 in SwAV helped mitigate the memory cost and only lead to a training time increase of 25 % 25\% compared to the classical setting using two crops of size 224 × 224 224\times 224 , while leading to a 4 point performance boost. As such, multi-crop is a very useful strategy to help boost performance for a marginal additional compute cost. It has thus become almost ubiquitous in recent works ( Caron et al., 2021 ; Zhou et al., 2022a ; Zhou et al., 2022b ; Bardes et al., 2022 ; Oquab et al., 2023 ) . It is worth pointing out that some works have only noticed minor increases in performance ( Wang et al., 2021a ) where it only lead to a 0.3 point performance increase.

Other approaches have emerged to negate the computational burden of feeding additional crops to the encoder by using nearest-neighbours in embedding space. While with NNCLR ( Dwibedi et al., 2021 ) the matched positive crop is replaced by its nearest-neighbour in latent space, in MSF ( Koohpayegani et al., 2021 ) , a k k -NN graph is built in embedding space to provide a similar effect as multi-crop and increase positive-pair related signal. This strategy was further employed in UniVCL ( Tang et al., 2022 ) which used augmentation strategies such as edge of node masking in combination with a k k -NN graph in latent space. All of these approaches show significant performance boosts for a smaller computational cost compared to multi-crop. In MSF, the use of this k k -NN graph only increases training time by 6%.

### 3.2 Role of the Projector

Most SSL with joint embedding methods include a projector (usually 2- or 3-layers MLP with ReLU) after the encoder. The SSL loss is applied to the projector’s output, and the projector is usually discarded after training. This crucial component was introduced in SimCLR ( Chen et al., 2020b ) and, although not responsible for avoiding collapse, allows significant top-1 accuracy gains on ImageNet. For example, in a 100-epochs training, the projector adds around 20 % 20\% of top-1 accuracy in SimCLR and VICReg (from around 50 % 50\% to 68 % 68\% and 48 % 48\% to 68 % 68\% respectively).

Bordes et al. (2022a) show that adding a projector is not only useful for SSL but is also highly beneficial in a supervised training setting when there is a misalignment between the training and downstream tasks (which was also demonstrated by Sariyildiz et al. (2022) ). In fact, it’s well known from Yosinski et al. (2014) that cutting layers in a trained deep neural network is beneficial when doing transfer learning mostly to avoid the training task’s overfitting bias. When looking through the lens of transfer learning, it becomes easy to understand why a projector is needed in SSL since the training task is always different from the downstream task. To bridge the gap between the terms used in the SSL and in the transfer learning literature, Bordes et al. (2022a) suggested coining the method of probing intermediate representations or cutting layers as: Guillotine Regularization (GR). They also highlight how crucial it is to dissociate GR from the addition of a projector in SSL because the optimal layer on which one should probe the representation might not always be the backbone (but could be an intermediate projector layer as demonstrated in Chen et al. (2020c) ). Lastly, Bordes et al. (2022a) demonstrated that reducing the misalignement between the training and pretext task (by using class label to find the positives pair in contrastive learning) leads to learning a network for which the best linear probe performance on ImageNet are obtained at the last projector layer (instead of the backbone) as shown in Figure 10 .

##### Using a projector to handle noisy image augmentations.

The projector may also be necessary to mitigate the noise of data augmentation. As described in Section 3.1 , SSL methods typically randomly augment input images to generate two different views of the same image. In some cases, enforcing invariance over two very different views might be a very strong constraint that could harm the performance, like when the content of the two views is different. To demonstrate how using the projector can mitigate that, we pretrain VICReg ( Bardes et al., 2021 ) with and without projector using image augmentations that are semantically similar according to an “oracle”, e.g a ResNet50 pretrained on ImageNet with full supervision. We pretrain for 100 100 epochs and include the linear probing results of these experiments in Table 2 . Without projector and with an oracle, the Top1 performance is 6.3 % 6.3\% higher compared to not using an oracle. However, equipped with a projector, using an oracle to remove noisy views only boosts Top1 performance by 0.6 % 0.6\% . This might imply that the projector has a role in handling inconsistent or noisy augmented views during the SSL training process.

##### Influence of the projector’s output dimension.

Similarly to how large batch sizes were seen as a requirement for contrastive methods, a large output dimension of the projector was seen as a requirement for covariance based methods. This is illustrated by figure 4 in Zbontar et al. (2021) , and table 12 in Bardes et al. (2021) , where drops of up to 15 % 15\% in top-1 on ImageNet can be observed. As pointed out in Garrido et al. (2022b) this was due to the projector’s intermediate layers scaling with the output dimension as well as loss weights that needed to be scaled as well. By tuning these parameters, VICReg’s top-1 accuracy increases from 55.9 % 55.9\% to 65.1 % 65.1\% with 256 dimensional embeddings. The peak performance is also achieved at 1024 dimensions and plateaus afterwards. While VICReg stays more sensitive to the output dimension of the projector than SimCLR, it is significantly more robust than originally thought and very large output dimensions are not a requirement. Comparable results should be achievable for Barlow Twins due to the similarities between the two methods.

##### Influence of the backbone’s output dimension.

Recent works also investigated the effect of the backbone dimension. Dubois et al. (2022) observed that larger backbone representations lead to better linear probe performance when using CISSL. Bordes et al. (2023b) investigated more deeply the impact of the backbone dimension across common SSL methods like VICReg, SimCLR or BYOL. They show that traditional supervised methods decline in performance when the dimension of the backbone is increased. On the other hand, SSL methods highly benefit from wider backbone representations as shown in Figure 12(a) . In fact, it is much more beneficial in SSL to increase the backbone size when training a ResNet than increasing the width or depth of the ResNet as illustrated in Figure 12(b) . This observation highlights that the current architectures used in SSL, which are often the same as the those used in supervised training, might not be optimal.

##### Properties of the representation induced by the projector.

Mialon et al. (2022) argue that the projector enforces pairwise independence of the features in the representation and provide a demonstration for random projectors in the context of VICReg, BarlowTwins and W-MSE ( Bardes et al., 2021 ; Zbontar et al., 2021 ; Ermolov et al., 2021 ) . In particular, higher degrees of independence are reached with wider projectors. Pairwise independence, or a soft notion thereof, can be more appropriate to learn unsupervised representations from “real world” datasets such as ImageNet than mutual independence ( Li et al., 2019 ) . Alternatively, seeking alternative SSL regularization to VCReg is needed if mutual independence is sought for. The optimization dynamics resulting from applying VCReg (the anti-collapse term in VICReg) at the projector’s output is also worth noting: minimizing VCReg with respect to the projector parameters is not necessary, and VCReg is rather optimized with respect to the encoder parameters. Whether this analysis fully extends to other SSL methods is an open question.

##### Training a SSL without a projector.

Jing et al. (2022) proposes DirectCLR, which shows that regularizing the representation in DirectCLR by applying the InfoNCE SimCLR objective on sub-vectors of the representation without a trainable projector is sufficient to outperform SimCLR with a linear projector in terms of ImagNet top-1 accuracy.

### 3.3 The Uniform Prior in SSL or the Failure of SSL on Unbalanced Data