A Cookbook of Self-Supervised Learning — part 8/8

Cleaned-text code points [109042, 119971). Bibliography omitted here; source.md includes it.

As a result, image encoders arising from such a joint pre-training are especially robust to visual changes that leave semantic meaning unchanged, such as sketches of objects as evaluated in ImageNet-Sketch ( Wang et al., 2019 ; Radford et al., 2021 ) , and are strong on out-of-domain generalization tasks. Yet, this is not always a desired representation, as visualizations in Ghiasi et al. (2022) show that these models also group features that are visually dissimilar, but semantically, or literally, alike. This can be mitigated, and overall performance, e.g. in linear probing, can even be improved by combining both image-text and image-image SSL as done in Mu et al. (2022) , who combine CLIP and SimCLR ( Radford et al., 2021 ; Chen et al., 2020b ) .

Recent work has pushed these vision-language systems to larger scales ( Ding et al., 2021 ; Yuan et al., 2021 ; Singh et al., 2022 ; Wang et al., 2022d ; Fang et al., 2022b ) , based on freely available image-caption pairs collected from the internet, such as in ( Schuhmann et al., 2022 ) . These modern SSL models are capable of representing both vision and text, and can be used in a number of applications that are multimodal, from visual-question answering to multimodal generation ( Alayrac et al., 2022 ; Li et al., 2022c ; Nichol et al., 2022 ; Rao et al., 2022 ) .

The future of vision-language pre-training, as an alternative to robust visual representations learned on vision alone, remains to be further explored. While its advantages in vision-language downstream applications are evident ( Shen et al., 2022 ; Dou et al., 2022 ) , shared embedding spaces can also be constructed by training solely the vision encoder first, fixing it, and then training a matching language encoder, as described in ( Zhai et al., 2022b ) . Ultimately, vision-language models are only the first step to self-supervised learning from multiple modalities at scale. Prototypes, such as Reed et al. (2022) , train self-supervised on arbitrary input streams, ranging from vision and text to tables and agent actions, and so learn re-usable representations that are helpful for general tasks.

### 4.3 Building Feature Extractors with Localization for Dense Prediction Tasks

Aside from semantic understanding, popular computer vision tasks from object detection to segmentation to depth estimation require models which extract localized features, in other words ones which contain information indicating the locations of objects within the input image. Self-supervised learning may be particularly valuable for these dense prediction tasks since collecting segmentation masks or bounding box annotations for training images is significantly more expensive than classification labels. However, learning frameworks which are carefully tuned on image classification benchmarks may lack traits which are valuable for such dense prediction tasks. Several works, which we note perform their experiments in different settings and on different architectures and learning algorithms, express seemingly contradictory findings, namely that existing self-supervised learning strategies are or are not effective for downstream dense prediction tasks ( Goyal et al., 2019 ; Purushwalkam and Gupta, 2020 ; Zhao et al., 2021 ; Ericsson et al., 2021b ; Shwartz-Ziv et al., ) . We now delve further into this discussion.

Limitations of self-supervised learners for localization. SSL approaches which rely on augmented views or jigsaw transformations, such as MoCo ( He et al., 2020b ) and PIRL ( Misra and Maaten, 2020 ) , learn occlusion invariance since they are trained with random crops on ImageNet where foreground objects are often large so that different crops contain different parts of the same object ( Purushwalkam and Gupta, 2020 ) . On the other hand, they lack viewpoint invariance and category-instance invariance. Further, Zhao et al. (2021) argue that self-supervised learners also lack localization information because the models are able to use all parts of the image, both foreground and background, to make their predictions. The above works conduct experiments principally on convolutional architectures. It is worth noting that Ericsson et al. (2021b) suggest that the best among the popular SSL algorithms they test on are CNNs, which can still achieve competitive performance with their supervised learning counterparts in some detection and segmentation settings. Interestingly, older pretext tasks such as jigsaw or colorization , which predate the recent SSL craze sparked by MoCo and SimCLR, can also achieve competitive performance compared to supervised learning backbones when the pretext task is made “hard” enough ( Goyal et al., 2019 ) .

CNNs or ViTs? Recent works suggest that vision transformers (ViTs) contain superior localization information in their learned representations compared to convolutional architectures ( Caron et al., 2021 ) . Whereas CNNs require specially designed segmentation pipelines to extract localization information from their features, this information arises naturally in the patchwise features of ViTs. Existing SSL methods designed specifically for transformers confirm that the trained models are effective for downstream detection and segmentation tasks, especially when fine-tuned ( Li et al., 2021b ; He et al., 2022 ) . However, it should be noted that these SSL algorithms explicitly demand localization in their objective functions, for example via masked autoencoding where patch features should contain information regarding the contents of the corresponding section of the image ( He et al., 2022 ) . More recently, masked autoencoding pre-training strategies have been adapted for convolutional architectures to great effect, where they achieve competitive performance on downstream object detection and instance segmentation ( Woo et al., 2023 ) . Moreover, we will see below that a variety of pre-training strategies designed specifically for localization can be effective on transformers and convolutional networks alike.

So how do we learn localized features without annotations? In order to tailor representations for downstream dense prediction tasks, numerous works propose modifying SSL routines specifically to enhance the localization in their features. Since these SSL pre-training algorithms do not use segmentation or detection annotations, they instead rely on carefully chosen unsupervised object priors.

One style of object prior enforces relationships between features extracted from locations within a single image, just as self-supervised learning procedures often enforce relationships between distinct images. One such prior uses the fact that adjacent ViT patches often contain the same objects. Unlike popular contrastive objectives which encourage augmented views of an image to produce similar features, SelfPatch encourages adjacent patches within a single image to produce similar features ( Yun et al., 2022 ) . A related method, DenseCL ( Wang et al., 2021c ) , matches the most similar pixel-wise features extracted from augmented samples to automatically handle the case in which augmentations move objects around in an image, and we only want to match features corresponding to the same object. More recently, VICRegL ( Bardes et al., 2022 ) applies a similar principle by combining geometric and learned matching, with a non-contrastive criterion. Just as clustering-based methods cluster related images, Leopart ( Ziegler and Asano, 2022 ) fine-tunes a pre-trained model to cluster patch-level features.

In addition to modifying the training loss to improve localization, we can also augment the data with this objective in mind by placing an object in multiple settings so that resulting models extract the same features from an object irrespective of its location. Instance Localization ( Yang et al., 2021 ) leverages RoIAlign ( He et al., 2017 ) , an algorithm designed for object detectors which extracts features corresponding to a specific image patch. To this end, Instance Localization pastes a randomly chosen patch cut from the foreground of one image onto two other images and extracts features corresponding to only the pasted foreground patch, using a contrastive loss to ensure that the foreground patch generates similar features regardless of the background present and regardless of its location within an image. A competing approach estimates the location of an object within the training image using saliency maps and then cuts and pastes these objects onto background and optimizes a similar objective ( Zhao et al., 2021 ) . Instead of using augmentations to move objects around, Purushwalkam and Gupta (2020) notes that nearby video frames contain the same object but in different positions or from different viewpoints so that contrastive learning on video data can serve much the same purpose.

Recently, UP-DETR ( Dai et al., 2021 ) and DETReg ( Bar et al., 2022 ) proposed an end-to-end SSL pretraining of the DETR family detectors. UP-DETR proposes to detect the bounding boxes of randomly selected patch regions in images conditioned on their pixel values while predicting their corresponding SwAV ( Caron et al., 2020 ) embedding. In DETReg, detection targets are obtained using the Selective Search algorithm, which does not require human annotations. Similarly, the detector predicts an associated SwAV ( Caron et al., 2018 ) embedding for each target bounding box.

Vision-language models for dense prediction tasks. In Section 4.2 , we saw that vision-language models extract semantically meaningful features. These features are also leveraged by recent works for open-vocabulary object detection ( Kamath et al., 2021 ; Gu et al., 2021 ; Zareian et al., 2021 ; Minderer et al., 2022 ) . These works leverage vision and language backbones pre-trained as previously discussed on captioned image databases and fine-tune on object detection data. Crucially, pre-trained language models, paired with image feature extractors, allow open-vocabulary object detectors to detect new objects never seen during their fine-tuning stage simply by querying the language model with an appropriate prompt.

## 5 Conclusion

Self-supervised learning (SSL) established a new paradigm for advancing machine intelligence. Despite many successes, SSL remains a daunting field with a dizzying array of methods each with intricate implementations. Due to the fast moving research and the breadth of SSL methods, it remains a challenge to navigate the field. This becomes an issue for researchers and practitioners who joined the field only recently, in turn creating a high barrier to entry for SSL research and deployment. We hope our cookbook will help lower these barriers by enabling the curious researcher of any background to navigate the terrain of methods, understand the role of the various knobs, and gain the know-how required to be successful with SSL.