A Cookbook of Self-Supervised Learning — part 7/8

Cleaned-text code points [93347, 109042). Bibliography omitted here; source.md includes it.

Applying SSL techniques to any of these data domains requires care as unique challenges arise in each domain which necessitate special considerations. For example, SSL for vision often revolves around data augmentations that may not naturally apply to speech signals. The ‘positive pairs’ available for contrastive learning varies from slightly different views of the same image to totally different segments of an audio recording. Nonetheless, both contrastive and generative objectives can be applied to these other data domains. One generically useful technique across data types is masking. Whether predicting missing words in a sentence, pixels in an image, or entries of a row in a table, masking is an effective component of SSL approaches across domains.

This section is not intended as a thorough survey of self-supervision for other data modalities, as each of those fields is vast. Domain-specific surveys can be found in Liu et al. (2022a) (audio), Schiappa et al. (2022b) (video), Min et al. (2021) (text), and Rubachev et al. (2022) (tabular data). Rather, this section provides a discussion of the interesting similarities and differences in how SSL is applied to audio, text, and tabular data.

Audio data. Audio signals, both raw audio and mel spectrograms, have a lot in common with images. As inputs to a neural network, there are strong similarities. For example convolutions can be useful ( Oord et al., 2016 ; Schneider et al., 2019 ; Baevski et al., 2021 ) . But as data for SSL, major differences arise. For example, horizontally flipping an image does not usually change the semantic meaning of an image (and is a wildly popular data augmentation), but for speech recordings this would completely distort the data. Similarly, while masking images is often done with random pixels, the two dimensions of a spectrogram represent time and frequency and masking with horizontal and/or vertical bands is more effective ( Wang et al., 2020b ) . Additionally, the existence of tones other than speech (background noise, room tone) presents a unique challenge when looking for positive pairs for contrastive learning, which is to prevent the learned representations from over fitting to the noise withing a given clip ( Oord et al., 2018 ; Wang et al., 2020b ) . In fact, the high frequency noisy artifacts, which are generally unrelated to the semantic meaning, mean reconstruction in input space is more complicated than other domains (e.g. text). Multi-modal models, on the other hand, can consider a soundbite and its text ( Sermanet et al., 2018 ; Chung et al., 2018 ) or some frames of a video and the corresponding sound clip ( Zhao et al., 2018 ; Alwassel et al., 2020 ) as different views to be used as positive pairs for contrastive learning.

Video data. Most of SSL images methods have a counter-part video SSL method. For instance, Feichtenhofer et al. (2021) have generalized SimCLR, MoCo, SwAV and BYOL to space-time video data. Indeed, in all these methods it is possible to incorporate the notion of similarity between different temporal clips of the same video. More recently, masked auto-encoding objectives for video have been built around the same idea as images, but by masking patches/ tubes of patches in the temporal axis as well ( Feichtenhofer et al., 2022 ; Tong et al., 2022 ; Girdhar et al., 2022 ) . Besides, it is common practice to use SSL vision pre-trained models for video downstream tasks like action recognition. With ViT for instance, the patch embedding convolutional layer can be transferred from 2D to 3D by repeating the weights along the temporal axis ( Feichtenhofer et al., 2022 ) . Vision models can then transfer to video models by using them as initialization for fine-tuning on video tasks ( Fang et al., 2022a ) . The frame features can also be used directly, by appending a linear layer on top of the features ( Radford et al., 2019 ) , or by using more complex heads ( Ni et al., 2022 ; Arnab et al., 2021 ) . In this case, the visual system is frozen and the temporal information is learned after.

Text data. In contrast to audio data, text is a relatively clean input signal and representations that are useful for reconstruction do not over fit to a noisy part of the signal. In fact, the most popular large language models are all trained with reconstruction objective as opposed to contrastive objectives popular in other data domains ( Radford et al., 2018 ; Radford et al., 2019 ; Brown et al., 2020 ; Devlin et al., 2018 ) . The Word2Vec objective ( Mikolov et al., 2013 ) predicts a masked out portion of the training text has served as a foundational objective for self-supervised learning in natural language. While uncommon, language modeling can be done with contrastive learning for word or or character representations ( Chen et al., 2022 ) . One other difference between text and images is that the masked token prediction for text is done over an entire dictionary. This approach is not the dominant one for images but it has been tried at the pixel level ( Chen et al., 2020a ) . While there are few augmentations for language data that do not change the semantic meaning, large scale systems generally use enough data and various types of masking to overcome this. Specifically, next token prediction ( Radford et al., 2018 ; Radford et al., 2019 ; Brown et al., 2020 ) is akin to masking the last token in a string, while bidirectional encoders mask tokens anywhere in the string ( Devlin et al., 2018 ) or fill larger spans of missing text ( Raffel et al., 2020 ; Tay et al., 2022 ) . This choice of unidirectional next-token prediction versus bidirectional approaches leads to meaningful differences in downstream text applications ( Artetxe et al., 2022 ) . For contrastive learning, positive pairs often come from masking and/or cropping input sequences ( Meng et al., 2021 ; Giorgi et al., 2021 ) . They can also be generated using dropout so that one input has two different latent representations ( Gao et al., 2021 ) . Additionally, some methods for both contrastive and reconstructive pretraining corrupt the input with several other augmentations including document rotation, sentence permutation, and token deletion ( Lewis et al., 2020 ; Raffel et al., 2020 ; Wu et al., 2018 ) .

Tabular data. Unlike text, audio, and images, classical machine learning tools are still popular for processing tabular data. However, while deep learning for tabular data is comparatively a small field, finding sensible data augmentation strategies is already a much studied topic. Several SSL methods for tabular data utilize masking in various ways and some techniques creatively employ other augmentations developed for images, like mixup ( Zhang et al., 2018 ) . As with images and audio, some algorithms aim to generate the missing or corrupted values while others employ contrastive learning. In combinatorial optimization such as Mixed Integer Programming (MIP), objective function is used as the guidance to generate positive solution pairs with comparable objectives and negative solution pairs whose objective values drastically differs despite tiny changes of a few variables ( Huang et al., 2023 ) . Similar approaches are also used in guided language generation ( Yang et al., 2023 ) .

The masked reconstruction approaches account for a variety of masking tactics. Furthermore, it is common with tabular data to predict mask vectors as a pretext task ( Yoon et al., 2020 ; Iida et al., 2021 ) . Since the predicting mask itself is part of the pretraining objective, the masked entries in the input must be filled, and typically this is done by sampling from the empirical distribution of that column or feature.

With the same augmentation, i.e. masking and sampling from the empirical marginal distribution, Bahri et al. (2021) propose pretraining with a contrastive loss. Specifically, they propose using the InfoNCE loss ( Gutmann and Hyvärinen, 2010 ; Ceylan and Gutmann, 2018 ) to compare the representations of the clean and corrupted inputs.

Several other works outline ways to augment the data for a combination of generation and contrastive learning. For example, tabular data can be split into groups of columns so each sample (row) has several views available Ucar et al. (2021) . Borrowing from vision systems, a combination of CutMix ( Yun et al., 2019 ) in input space and mixup ( Zhang et al., 2018 ) in embedding space is also an effective augmentation for tabular data ( Somepalli et al., 2021 ) . These methods generate augmented views that are used along with the clean input for contrastive learning. However, contrastive pretraining for both the SAINT model ( Somepalli et al., 2021 ) and SubTab ( Ucar et al., 2021 ) seems to work best when this is paired with a reconstructive loss term.

In their work focusing on comparing the SSL methods for tabular data, Rubachev et al. (2022) find that pretraining objectives generally do help boost the performance of tabular models. But more specifically, they find that pretraining objectives that use the labels are best, implying that SSL for tabular data has yet to be the state of the art in its domain ( Rubachev et al., 2022 ) . Similarly, Levin et al. (2023) show that unlike in computer vision, existing SSL pre-training routines yield less transferable features than supervised pre-training.

Reinforcement learning. SSL has been used to improve reinforcement learning (RL) on visual inputs. This setting is similar to video, except apart from the sequence of images, we also have access to the sequence of actions. The most common approach to apply SSL here is to use contrastive learning to train a model to match current state representation and the next time step’s representation, or to match representations of the same state but with different augmentations applied. One of the earliest examples is CURL ( Srinivas et al., 2020 ) . Recently, SSL has been used to improve sample efficiency on a challenging Atari100k benchmark ( Kaiser et al., 2020 ) . Recent works have modified BYOL ( Grill et al., 2020 ) or Barlow Twins ( Zbontar et al., 2021 ) by feeding images of consecutive timesteps’ observations to the two branches of the siamese network: SGI ( Schwarzer et al., 2021b ) and Barlow Balance ( Zhang et al., 2022c ) did this for offline pretraining, while SPR ( Schwarzer et al., 2021a ) uses it as an additional objective in the online setting. The best-performing method doing this is EfficientZero ( Ye et al., 2021 ) , which modifies MuZero ( Schrittwieser et al., 2020 ) by, among other modifications, adding the SimSiam ( Chen and He, 2020 ) objective to train the encoder and the forward model, and sets the new state of the art on Atari100k. Parisi et al. (2022) propose PVR, a method based on MoCo ( He et al., 2020a ) that improves sample efficiency on control tasks. Eysenbach et al. (2022) show that contrastive learning in RL setting is directly linked to goal-conditioned RL, and demonstrate that a method based on InfoNCE ( Oord et al., 2018 ) achieves great performance on robotic arm control tasks.

SSL has been shown to yield good representations for behavior cloning. Pari et al. (2022) show that imagenet-pretrained model finetuned with BYOL ( Grill et al., 2020 ) can be very effectively used for visual imitation on robotic open, push and stack tasks, while Arunachalam et al. (2022) use a similar method and successfully learn from a small manipulation dataset collected using VR. Guzey et al. (2023) present a method that uses BYOL to extract information from tactile sensors on robotic arms and improve dexterous manipulation. Cui et al. (2022) shows that BYOL representations of visual inputs are also useful when modeling goal-conditioned trajectories with transformer architecture.

There are a few additional challenges when applying SSL to RL. First, if the data is recorded on-line, individual observations are highly correlated with each other and are not IID (independent and identically distributed), so sampling from replay buffer should be done carefully. One failure mode of SSL objectives when applied to RL agents’ data is the proclivity to latch on ‘slow features’ ( Sobal et al., 2022 ) . The contrastive objective may learn for example to only look at the cloud patterns in the sky to tell apart frames in a self-driving dataset, so one must be careful to design augmentations in a way to remove useless static features in the image, or to sample data accordingly.

SSL has been used not only to improve sample efficiency, but also to improve exploration. Guo et al. (2022b) propose BYOL-Explore which uses BYOL ( Grill et al., 2020 ) to learn the encoder and the forward model, and use the forward model disagreement as the exploration objective. The follow-up work by Jarrett et al. (2022) address the problem of BYOL-Explore latching on a noisy TV. Yarats et al. (2021) proposed using a clustering method akin to SwAV ( Caron et al., 2020 ) to do unsupervised exploration, i.e. exploration with only intrinsic rewards.

A few works have explored using vast natural videos data available to pre-train representations for RL agents. Xiao et al. (2022) introduce MVP, which uses masked-autoencoder to pre-train the transformer encoder for robotic control, while Ma et al. (2022) propose VIP, a method to learn universal features for RL using ResNet-50 backbone and the objective based the time between frames in the observations as the supervision signal. Another method for training foundation models for RL, R3M ( Nair et al., 2022 ) , combines time-contrastive and video-language alignment objectives. VIP and R3M are trained on the large Ego4D dataset ( Grauman et al., 2022 ) , while MVP combines Imagenet, Ego4D, and additional hand manipulation data. Majumdar et al. () propose VC-1, a method based on masked auto-encoding. The authors test the proposed method and other foundation models on the new test suite called CortexBench. The benchmark includes control, object manipulation, and navigation tasks, with different methods excelling at different parts of the benchmark.

There are also unsupervised methods for learning representations that are specific to RL and are not commonly used for images: e.g. Laplacian eigenmaps ( Machado et al., 2017 ) , forward-backward representations ( Touati et al., 2023 ) . Zhang et al. (2021) propose to learn representations by making representations the same for states that lead to the same rewards, and different otherwise.

### 4.2 Incorporating Multiple Modalities into SSL Training

Self-supervised learning need not be based on a single modality. Especially multi-modal vision-language have recently demonstrated this to great effect. Contrastive Language–Image Pre-training (CLIP) ( Radford et al., 2021 ) , and ALIGN ( Jia et al., 2021 ) are self-supervised learning approaches that use image-caption pairs to learn a joint embedding space for images and captions. The objective here is contrastive, given an image and its caption are fed through separate encoder models that encode each modality into a fixed-length embedding vector. The embeddings of the training data image-caption pair are aligned, whereas other combinations in a batch are repelled.

This approach is especially interesting in comparison to contrastive SSL based on pure vision, as discussed in Section 2.6.1 . The use of a second modality, here text, anchors the entire SSL training. It is no longer necessary to generate multiple augmented views to form a notion of robust representation as the joint approach learns semantically meaningful representations simply by observing similar captions re-occurring with similar images.