A Cookbook of Self-Supervised Learning — part 3/8
Cleaned-text code points [31124, 46113). Bibliography omitted here; source.md includes it.
These ideas were extended to deep learning in Deep Canonically Correlated Autoencoders (DCCAE) an autoencoder regularized via CCA. Hsieh (2000) and Andrew et al. (2013) introduce the objective of jointly learning parameters for two networks, f 1 , f 2 f_{1},f_{2} , such they their outputs are maximally correlated. The inputs to these networks are two views X 1 X_{1} and X 2 X_{2} . Specifically the objective is then to find parameters θ 1 , θ 2 \theta_{1},\theta_{2} for each network such that ( θ 1 ∗ , θ 2 ∗ ) = argmax ( θ 1 , θ 2 ) corr ( f 1 ( X 1 , θ 1 ) , f 2 ( X 2 , θ 2 ) CLOSE . \displaystyle(\theta_{1}^{*},\theta_{2}^{*})=\text{argmax}_{(\theta_{1},\theta_{2})}\text{corr}(f_{1}(X_{1};\theta_{1}),f_{2}(X_{2};\theta_{2}). (14)
This DCCAE objective was extended to multivariate outputs and arbitrary DDNs in Wang et al. (2015) .
From these origins, stems SSL methods such as VICReg ( Bardes et al., 2021 ) , Barlow Twins ( Zbontar et al., 2021 ) , SWAV ( Caron et al., 2020 ) , and W-MSE ( Ermolov et al., 2021 ) . VICReg , the most recent among these methods, balances three objectives based on co-variance matrices of representations from two views: variance, invariance, co-variance shown in Figure 6 . Regularizing the variance along each dimension of the representation prevents collapse, the invariance ensures two views are encoded similarly, and the co-variance encourages different dimensions of the representation to capture different features.
### 2.5 Masked Image Modeling
A number of prominent early self-supervised pre-training algorithms for computer vision applied degradations to training images, such as decolorization ( Zhang et al., 2016 ) , noise ( Vincent et al., 2008 ) , or shuffling image patches ( Noroozi and Favaro, 2016 ) , and taught models to undo these degradations. Context encoders instead mask out large portions of an image and replace their pixel values with white, teaching an autoencoder to inpaint the white patches ( Pathak et al., 2016 ) . This early attempt at masked image modeling does not achieve competitive performance with supervised learning on downstream tasks, and pre-dates vision transformer architectures which modern masked training routines build upon. Subsequently, BERT ( Devlin et al., 2019 ) shook up the natural language processing world by replacing text tokens input to a transformer language model with learnable mask tokens and teaching the model to recover the original text. This paradigm, termed masked language modeling (MLM), can also be interpreted as a form of the above strategy, degrading a sample via masking and teaching a model to undo the masking degradation. MLM, along with span-infilling techniques, remains popular as a SSL objective for large language models ( Raffel et al., 2020 ; Wang et al., 2022a ; Tay et al., 2022 ) .
We can also similarly mask out portions of an image and teach a model to inpaint them. This pre-training vision strategy is known as masked image modeling (MIM). Inspired by BERT, Dosovitskiy et al. () exploit the vision transformer architecture by masking out patch tokens and replacing them with learned mask tokens. They then teach their model to predict pixel values directly, but they find that this pre-training strategy is significantly less effective than supervised pre-training.
Bao et al. (2021a) note that applying the BERT strategy directly to images is difficult because whereas text tokens can only take on a small number of values that can be predicted as a classification problem, image patches can assume considerably more possible values and hence more classes than would be suitable for classification. Instead, the authors cast MIM as a regression problem, first using an autoencoder to encode image patches as discrete tokens, and then pre-training their transformer to predict the discrete token values for masked tokens. BEiT achieves significantly improved performance on downstream image classification and semantic segmentation over previous supervised and self-supervised baselines, but its training pipeline is complex since it requires a powerful autoencoder for converting image patches to discrete tokens.
In order to streamline MIM pre-training, two concurrent works ( He et al., 2022 ; Xie et al., 2022 ) propose simplified algorithms, masked autoencoders (MAE) and SimMIM respectively, which directly reconstruct masked image patches rather than discrete image tokens extracted from an encoder as in BEiT. Moreover, these simplified pre-training strategies achieve superior performance to BEiT on downstream image classification, semantic segmentation, and object detection tasks. Since then, masked image modeling has achieved competitive performance on a wide variety of vision tasks ( Zhou et al., 2022a ; Woo et al., 2023 ; Oquab et al., 2023 ) and even vision-language representation learning ( Fang et al., 2022a ) . The most successful approaches when using a frozen encoder, iBOT ( Zhou et al., 2022a ) and DINOV2 ( Oquab et al., 2023 ) employ a mix of masked image modeling and more classical approaches such as self-distillation. Howver, their masked image modeling objective reconstructs in latent-space with a teacher network used to provide targets instead of using the original image as the reconstruction target.
Consider that MIM is fundamentally a generative modeling task. Such models are trained to generate missing image parts conditional on the observed ones. Note that BEiT, MAE, and SimMIM are deployed on downstream prediction problems by removing the decoder and replacing it with a prediction head. However, masked image models can also achieve strong generative modeling ( Chang et al., 2022 ) , including text-conditional generation ( Chang et al., 2023 ) . Compared to autoregressive models for image generation ( Yu et al., ) which generate patches sequentially, MIM-based generative models are significantly more efficient, since they can generate patches in parallel.
In Section 3.6 , we will discuss various techniques harnessed by state-of-the-art masked image modeling systems to achieve such competitive performance.
### 2.6 A Theoretical Unification Of Self-Supervised Learning
#### 2.6.1 Theoretical Study of SSL
Numerous works have attempted to unify various SSL methods. In Huang et al. (2021) , Barlow Twins’ criterion is shown to be linked to an upper bound of a contrastive loss. This suggests a link exists between contrastive and covariance-based methods. This direction was further pursued in Garrido et al. (2022b) , where a covariance-based and contrastive criterion are shown to be equivalent up to normalization by deriving the precise gap between the two approaches. These results were further validated empirically as methods were shown to exhibit similar performance and representation properties at ImageNet’s scale (1.2 million samples). The similarities among methods was also studied in Tao et al. (2021) where this unification was tackled from a study of the losses’ gradients.
##### Relationship between Contrastive Learning and Other Objectives.
Initially, InfoNCE was suggested as a variational approximation to the mutual information between two views ( Aitchison and Ganev, 2023 ; Wang and Isola, 2020 ; Oord et al., 2018 ) . Li et al. (2021a) explains the role of InfoNCE in contrastive learning through the lens of the Hilbert-Schmidt Independence Criterion (HSIC), which was used to present a variational lower bound on the mutual information (MI) between different transformations. Tschannen et al. (2020) shows the performance of InfoNCE cannot be explained only in terms of mutual information. Instead other factors such as the feature extractor and formualtion of the mutual information estimator are important and can lead to drastically different performance ( Guo et al., 2022a ) . Alternative theories suggest that InfoNCE balances alignment of “positive” examples and uniformity of the overall feature representation ( Wang and Isola, 2022 ) , or that (under strong assumptions) it can identify the latent structure in a hypothesized data-generating process, akin to nonlinear ICA ( Khemakhem et al., 2020 ) . In Wang and Isola (2020) , Theorem 1 shows that contrastive learning with an RBF kernel (an expressive map of features into a higher dimensional space) converges to a uniform distribution on the sphere with matched pairs. ( Tian, 2022 ) shows that contrastive learning with deep linear network is equivalent to Principal Component Analysis (PCA) and ( Tian, 2023 ) further analyzes the role played by nonlinearity in the architecture if trained with contrastive loss, showing that nonlinearity leads to many local optima that can host diverse patterns in the training data, while linear networks only allow a single dominant pattern to be learned. Hjelm et al. (2019) introduced Deep InfoMax (DIM), which maximizes the mutual information between the input and output of a deep neural network encoder using local features from the input, an idea that was extended to graphs in Veličković et al. (2018) .
##### Unified contrastive losses.
Tian (2022) unified contrastive losses as minimizing a general family of loss functions ℒ ϕ , ψ \mathcal{L}_{\phi,\psi} , where ϕ \phi and ψ \psi are monotonously increasing and differentiable scalar functions min 𝜽 ℒ ϕ , ψ ( 𝜽 ) = ∑ i = 1 N ϕ ( ∑ j ≠ i ψ ( ‖ 𝒛 i − 𝒛 i ′ ‖ 2 2 − ‖ 𝒛 i − 𝒛 j ‖ 2 2 ) ) . \min_{{\bm{\theta}}}\mathcal{L}_{\phi,\psi}({\bm{\theta}})=\sum_{i=1}^{N}\phi\left(\sum_{j\neq i}\psi(\|{\bm{z}}_{i}-{\bm{z}}_{i^{\prime}}\|_{2}^{2}-\|{\bm{z}}_{i}-{\bm{z}}_{j}\|_{2}^{2})\right). (15) where z z are representations with indices i i and j j running from 1 1 to N N . With different ϕ \phi and ψ \psi , Eqn. 15 covers many loss functions ( Figure 8 ). In particular, setting ϕ ( x ) = τ log ( ϵ + x ) \phi(x)=\tau\log(\epsilon+x) and ψ ( x ) = exp ( x / τ ) \psi(x)=\exp(x/\tau) gives a generalized version of InfoNCE loss ( Oord et al., 2018 ) : ℒ n c e := − τ ∑ i = 1 N log e − ∥ 𝒛 i − 𝒛 i ′ ∥ 2 2 / τ ϵ e − ∥ 𝒛 i − 𝒛 i ′ ∥ 2 2 / τ + ∑ j ≠ i e − ∥ 𝒛 i − 𝒛 j ∥ 2 2 / τ \!\!\mathcal{L}_{nce}\!:=\!-\tau\sum_{i=1}^{N}\log\frac{e^{-\|{\bm{z}}_{i}-{\bm{z}}_{i^{\prime}}\|_{2}^{2}/\tau}}{\epsilon e^{-\|{\bm{z}}_{i}-{\bm{z}}_{i^{\prime}}\|_{2}^{2}/\tau}+\sum_{j\neq i}e^{-\|{\bm{z}}_{i}-{\bm{z}}_{j}\|_{2}^{2}/\tau}} (16) where ϵ > 0 \epsilon>0 is some constant e.g. ϵ = 1 \epsilon=1 has been used in He et al. (2020b) ; Tian et al. (2020a) , ϵ = 0 \epsilon=0 yields a slight variation of SimCLR ( Chen et al., 2020b ) , the DCL loss ( Yeh et al., 2021 ) .
##### Hard negative sampling.
Negative mining has been thoroughly studied in (deep) metric learning. Recently, some works have focused on putting more weight on hard samples ( Robinson et al., 2020 ) . Yet, Kalantidis et al. (2020) ; Tian (2022) showed that contrastive SSL losses with ψ = e x / τ \psi=e^{x/\tau} already have such mechanisms at the batch level, focusing on hard-negative pairs without explicit "hard-negative sampling". This means that contrastive losses need large batch sizes to ensure that hard negative samples are observed which occurs at an additional memory cost.
##### Study of the projector.
The projector network, first introduction by Chen et al. (2020b) , maps the representations into another space where the loss is computed. Despite strong empricial evidence the projector improves performance, few theoretical works attempted to explain its role. Jing et al. (2022) study the role of linear projectors in contrastive learning. More precisely, it is argued that the projector prevents dimensional collapse in the representation space and that it only needs to be diagonal and low-rank to do so. Although the proposed method without a projector outperforms SimCLR with a one layer linear projector, for 2- and 3-MLP projectors, performance remains out of reach. Cosentino et al. (2022) study the interplay of the projector and data augmentations when the augmentations are Lie group transformations, and, as Mialon et al. (2022) , provide an explanation on the effect of width and depth of the projector. Further empirical investigations of the role of the projector are presented in section 3.2 .
#### 2.6.2 Dimensional Collapse of Representations
While the goal of joint self-supervised methods is to learn meaningful representations, a significant part of the approaches suffer from what is called dimensional collapse . Dimensional collapse occurs when information encoded across different dimensions of the representation is redundant. In other words in the output of the projector, the embeddings are rank-deficient, which can be approximated via the singular value spectrum of the embeddings, as illustrated in Figure 12 .
This phenomenon was first illustrated by Hua et al. (2021) where the use of a whitening batch normalization helped alleviate collapse. Dimensional collapse was also studied from a theoretical point of view by Jing et al. (2022) with a focus on contrastive methods. Several following works linked dimensional collapse to an impact on performance ( He and Ozay, 2022 ; Ghosh et al., 2022 ; Li et al., 2022a ; Garrido et al., 2022a ) . Some works focused on unsupervised evaluation ( Ghosh et al., 2022 ; Garrido et al., 2022a ) where dimensional collapse was found to be a good proxy for downstream performance. Different measures of dimensional collapse have been introduced such as the entropy of the singular value distribution ( Garrido et al., 2022a ) , the classical rank estimator ( Jing et al., 2022 ) , fitting a power law to the singular value distribution ( Ghosh et al., 2022 ) or the AUC of the singular value distribution ( Li et al., 2022a ) . Nonetheless, all of these measures focus on evaluating the rank of the representations to measure dimensional collapse in the learned representations.
### 2.7 Pretraining Data
##### Curated (standard)
: The most common practice is to pretrain SSL models on curated datasets such as ImageNet and alternatives such as PASS Asano et al. (2021) . These datasets tend to generally be class-balanced and contain object-centric images, where the object is prominentely feature often in the center of the photo.
##### Training with data from the wild
: Even though ImageNet has been the dataset of choice for pretraining, it is definitely not the only option. Its simplicity (object centric, single object, balanced classes) makes it a very good playground but most datasets in the wild are not as clean. If we want to leverage large uncurated datasets for SSL methods need to translate well outside of ImageNet. To this effect some works have explored pretraining on large uncurated datasets ( Goyal et al., 2021 ) , or on datasets that different from ImageNet such as COCO ( El-Nouby et al., 2021 ) , or iNaturalist ( Assran et al., 2022a ) . While these works have shown promising results, ImageNet (or similarly curated dataset) pretraining has remained the norm.