A Cookbook of Self-Supervised Learning — part 1/8

Cleaned-text code points [0, 15416). Bibliography omitted here; source.md includes it.

# A Cookbook of Self-Supervised Learning

## 1 What is Self-Supervised Learning and Why Bother?

Self-supervised learning , dubbed “the dark matter of intelligence” 1 1 1 https://ai.facebook.com/blog/self-supervised-learning-the-dark-matter-of-intelligence/ , is a promising path to advance machine learning. As opposed to supervised learning , which is limited by the availability of labeled data, self-supervised approaches can learn from vast unlabeled data ( Chen et al., 2020b ; Misra and Maaten, 2020 ) . Self-supervised learning (SSL) underpins deep learning’s success in natural language processing leading to advances from automated machine translation to large language models trained on web-scale corpora of unlabeled text ( Brown et al., 2020 ; Popel et al., 2020 ) . In computer vision, SSL pushed new bounds on data size with models such as SEER trained on 1 billion images ( Goyal et al., 2021 ) . SSL methods for computer vision have been able to match or in some cases surpass models trained on labeled data, even on highly competitive benchmarks like ImageNet ( Tomasev et al., 2022 ; He et al., 2020a ; Deng et al., 2009 ) . SSL has also been successfully applied across other modalities such as video, audio, and time series ( Wickstrøm et al., 2022 ; Liu et al., 2022a ; Schiappa et al., 2022a ) .

Self-supervised learning defines a pretext task based on unlabeled inputs to produce descriptive and intelligible representations ( Hastie et al., 2009 ; Goodfellow et al., 2016 ) . In natural language, a common SSL objective is to mask a word in the text and predict the surrounding words. This objective of predicting the context surrounding a word encourages the model to capture relationships among words in the text without the need for any labels. The same SSL model representations can be used across a range of downstream tasks such as translating text across languages, summarizing, or even generating text, along with many others. In computer vision, analogous objectives exist with models such as MAE or BYOL learning to predict masked patches of an image or representation ( Grill et al., 2020 ; He et al., 2022 ) . Other SSL objectives encourage two views of the same image, formed by say adding color or cropping, to be mapped to similar representations.

With the power to train on vast unlabeled data comes many benefits. While traditional supervised learning methods are trained on a specific task often known a priori based on the available labeled data, SSL learns generic representations useful across many tasks. SSL can be especially useful in domains such as medicine where labels are costly or the specific task can not be known a priori ( Krishnan et al., 2022 ; Ciga et al., 2022 ) . There’s also evidence SSL models can learn representations that are more robust to adversarial examples, label corruption, and input perturbations—and are more fair—compared to their supervised counterparts ( Hendrycks et al., 2019 ; Goyal et al., 2022 ) . Consequently, SSL is a field garnering growing interest. Yet, much like cooking, training SSL methods is a delicate art with a high barrier to entry.

### 1.1 Why a Cookbook for Self-Supervised Learning?

While many components of SSL are familiar to researchers, successfully training a SSL method involves a dizzying set of choices from the pretext tasks to training hyper-parameters. SSL research has a high barrier to entry due to (i) its computational cost, (ii) the absence of fully transparent papers detailing the intricate implementations required to fully enable SSL’s potential, and (iii) the absence of a unified vocabulary and theoretical view of SSL. As SSL established a distinct paradigm from traditional reconstruction-based unsupervised learning methods such as (denoising, variational) Autoencoders ( Vincent et al., 2008 ; Vincent et al., 2010 ; Kingma and Welling, 2013 ) , our vocabulary for understanding SSL in a unified view is limited. In fact, attempts at unifying SSL methods under a single viewpoint have only started to emerge in the last year ( HaoChen et al., 2021 ; Balestriero and LeCun, 2022 ; Shwartz-Ziv et al., 2022 ; Garrido et al., 2022b ) . Without a common ground to characterize the different components of SSL methods, it’s more challenging for researchers to start working on SSL methods. Meanwhile, SSL research is in dire need for new researchers since SSL is now deployed throughout the real-world. Yet, many open research questions remain regarding SSL’s generalization guarantees, fairness properties, and robustness to adversarial attacks or even naturally occurring variations. Such questions are crucial to the reliability of SSL methods.

Furthermore, SSL—which is empirically driven—comes with many moving pieces (mostly hyper-parameters) that may impact key properties of the final representations and are not necessarily well-detailed in published work. That is, to start studying SSL methods, one must first exhaustively empirically probe those methods to fully grasp the impact and behaviors of all those moving pieces. Such empirical blind spots are strong limitations as they demand large computational resources and pre-existing hands-on experience. All in all, the co-occurrence of SOTA performances from seemingly different yet overlapping methods, little existing theoretical research, and widespread real-world deployment, make the need for a cookbook unifying the techniques and their recipes essential to lower SSL’s research barrier to entry.

Our goal is to lower the barrier to entry into SSL research by laying the foundations and latest SSL recipes in the style of a cookbook. To successfully cook, you must first learn the basic techniques: chopping, sautéing, etc. We begin in Section 2 with the fundamental techniques of self-supervised learning using a common vocabulary. Specifically, we describe the families of methods along with theoretical threads to connect their objectives in a unified perspective. We highlight key concepts such as loss terms or training objectives in concept boxes. Next, a cook must learn to skillfully apply the techniques to form a delicious dish. This requires learning existing recipes, assembling ingredients, and evaluating the dish. In Section 3 we introduce the practical considerations to implementing SSL methods successfully. We discuss common training recipes including hyperparameter choices, how to assemble components such as architectures and optimizers, as well as how to evaluate SSL methods. We also share practical tips from leading researchers on common training configurations and pitfalls. We hope this cookbook serves as a practical foundation for successfully training and exploring self-supervised learning.

## 2 The Families and Origins of SSL

SSL methods have enjoyed a renaissance since 2020, thanks in large part to the availability of extremely large datasets and high-memory GPUs. However, the origins of SSL go back to the very beginning of the deep learning era.

### 2.1 Origins of SSL

Contemporary methods build upon the knowledge we gained from early experiments. In this section, we give a brief overview of the main ideas of SSL prior to 2020. While many of the specific methods have fallen out of mainstream use because they no longer provide state-of-the-art performance on benchmark problems, and they will not be discussed in great detail, the ideas from these papers form the foundation for many of the modern methods. For example, the core objective of restoring missing or distorted parts of an input or contrasting two views of the same image form the foundation for modern SSL methods. Early progress in SSL focused on the development of methods that fell into the following (sometimes overlapping) categories:

1. Information restoration: A wide range of methods have been developed that mask or remove something from an image, and then train a neural network to restore the missing information. Colorization-based SSL methods convert an image to grayscale, and then train a network to predict the original RGB values ( Zhang et al., 2016 ; Larsson et al., 2016 ; Vondrick et al., 2018 ) . Because colorization requires understanding object semantics and boundaries, colorization was demonstrated as an early SSL method for object segmentation. The most straightforward application of information restoration is to mask, aka remove, a portion of an image and then train a network to inpaint the missing pixel values ( Pathak et al., 2016 ) . This idea evolved into masked auto-encoding methods ( He et al., 2022 ) , in which the masked region is a union of image patches that can be predicted using a transformer.

2. Using temporal relationships in video: While the focus of this review is on image (and not video) processing, a range of specialized methods have been developed for learning single-image representations by pre-training on videos. Note that information restoration methods are particularly useful for videos, which contain multiple modalities of information that can be masked. Wang and Gupta (2015) pre-train a model using a triplet loss that promotes similarities between representations of an object in two different frames. The resulting model performed well for object detection. Pathak et al. (2017) trains a model to predict the motion of objects in a single frame, and adapts the resulting features to solve single-frame detection problems. Agrawal et al. (2015) predicts the ego-motion of a camera given multiple frames. Owens et al. (2016) propose to remove the audio track from a video, and then predict the missing sound. For specialized applications like depth mapping, self-supervised methods have been proposed that learn monocular depth models from unlabeled image pairs ( Eigen et al., 2014 ) and later the frames from a single-camera video ( Zhou et al., 2017 ) . Such methods remain an active area of research.

3. Learning spatial context: This category of methods trains a model to understand the relative positions and orientations of objects within a scene. RotNet ( Gidaris et al., 2018 ) masks the direction of gravity by applying a random rotation and then asks the model to predict the rotation. Doersch et al. (2015) is one of the first SSL methods that simply predicts the relative location of two randomly sampled patches in an image. This strategy was superseded by “jigsaw” methods ( Pathak et al., 2016 ; Noroozi et al., 2018 ) that break an image into an array of disjoint patches and predict the relative location of each. A different spatial task is learning to count ( Noroozi et al., 2017 ) : the model is trained to output the number of objects in an image in a self-supervised way.

4. Grouping similar images together: One can learn rich features by grouping semantically similar images together. K-means clustering is one of the most widely used methods from classical machine learning. A number of studies have adapted k-means to perform SSL with neural models. Deep clustering alternates between assigning labels to images by performing k-means in the feature space, and updating the model to respect these assigned class labels ( Caron et al., 2018 ) . More recent treatments of this approach use mean-shift updates to push features towards their cluster center, and have been shown to complement BYOL, a method based on two networks with the objective to predict pseudo-labels for each sample ( Koohpayegani et al., 2021 ) (discussed in Section 2.3 ). Other improvements to deep clustering include using optimal transport methods in feature space to create more informative clusters ( Asano et al., 2019 ) .

5. Generative models: An early influential SSL method is greedy layer-wise pretraining ( Bengio et al., 2006 ) , in which layers of a deep network are trained one-at-a-time using an autoencoder loss. An analogous approach from the time used Restricted Boltzman Machines (RBMs), which could be trained layer-wise and stacked to create deep belief nets ( Hinton et al., 2006 ) . While these methods were abandoned in favor of simpler initialization strategies and longer training runs, they were historically impactful uses of SSL, as they enabled the training of the first “deep” networks. Later advancements improved on the representation learning ability of auto-encoders, including denoising autoencoders ( Vincent et al., 2008 ) , cross-channel prediction ( Zhang et al., 2017 ) , and deep canonically correlated autoencoders ( Wang et al., 2015 ) . Nonetheless, it was ultimately found that representation transferability is better when the auto-encoder is asked to restore a missing part of its input, resulting in the “information restoration” category of SSL methods.

Generative Adversarial Networks (GANs) ( Goodfellow et al., 2014 ) consist of an image generator and a discriminator that differentiates real images from generated images. Both components of this model pair can be trained without supervision, and both potentially contain knowledge useful for transfer learning. Early GANs papers ( Salimans et al., 2016 ) experimented with downstream image classification using GAN components. Specialized feature learning routines have also been developed that modify the discriminator ( Springenberg, 2015 ) , add a generator ( Dai et al., 2017 ) , or learn additional mappings from image to latent space ( Donahue et al., 2017 ) to improve transfer learning.

6. Multi-view invariance: Many modern SSL methods, especially those that we focus on in this article, use contrastive learning to create feature representations that are invariant to simple transforms. The idea of contrastive learning is to encourage a model to represent two augmented versions of an input similarly. A number of methods led the charge in this direction by enforcing invariance in various ways before contrastive learning was widely adopted.

One of the most popular frameworks for learning from unlabeled data is to use a weakly trained network to apply pseudolabels to images, and then train using these labels in a standard supervised fashion ( Lee et al., 2013 ) . This approach was later improved by enforcing invariance to transformations. Virtual adversarial training ( Miyato et al., 2018 ) trains a network on images using their pseudolabels, and additionally performs adversarial training so that learned features are nearly invariant to small perturbations to the input image. Later works focused on maintaining invariance to data augmentation transforms. Important early methods in this category include MixMatch ( Berthelot et al., 2019 ) , which chooses pseudolabels by averaging outputs of a network on several different random augmentations of the training images, resulting in labels that are augmentation invariant. Around the same time, it was discovered that good SSL performance could be achieved by training a network to maximize the mutual information between the representations of an image under different views ( Bachman et al., 2019 ) . These augmentation-based methods formed a bridge between the older methods described above and the contemporary methods that are the focus of this paper.

With these origins, we now turn to categorizing SSL into four broad families: The Deep Metric Learning Family, The Self-Distillation Family, The Canonical Correlation Analysis Family, and the Masked Image Modeling Family.

### 2.2 The Deep Metric Learning Family: SimCLR/NNCLR/MeanSHIFT/SCL