A Cookbook of Self-Supervised Learning — part 2/8

Cleaned-text code points [15416, 31124). Bibliography omitted here; source.md includes it.

The Deep Metric Learning (DML) family of methods is based on the principle of encouraging similarity between semantically transformed versions of an input. DML originated with the idea of a contrastive loss , which transforms this principle into a learning objective. Contrastive loss was first introduced in ( Bromley et al., 1993 ) then more formally defined in ( Chopra et al., 2005 ; Hadsell et al., 2006 ) . In DML one trains a network to predict whether two inputs are from the same class (or not) by making their embedding close (or far from each other). Since data is without labels, to identify similar inputs, we often form variants of a single input using known semantic preserving transformations. The variants of the inputs are called positive pairs or examples; the samples we wish to make dissimilar are called negatives . Often there’s a margin parameter, m m , imposing the distance between examples from different classes should be larger than m m . Similar to the contrastive loss, the Triplet loss ( Weinberger and Saul, 2009 ; Chechik et al., 2010 ; Schroff et al., 2015 ) shares a similar spirit, but is composed of triplets: a query, a positive example, and a negative example (see eq. 3 ). Compared to contrastive loss, triplet loss only requires the difference of (dis-)similarities between positive and negative examples to the query point to be larger than a margin m m .

The shift from DML to what is now referred to as SSL might have occurred when Sohn (2016) introduced the (N+1)-tuple loss, a loss similar to the contrastive predictive coding (CPC) loss from ( Oord et al., 2018 ) . The use of other sample positive views as the negative view of other pairs is introduce as an efficient strategy coined N-pair-mc loss . Ni et al. (2021b) shows that contrastive learning is a special case of meta-learning, and existing meta-learners can be directly applied to SSL with competitive performance. CPC was extended to images in ( Henaff, 2020 ) . A key ingredient in CPC was the introduction of the InfoNCE loss described in 3 Oh Song et al. (2016) , which became central in SSL.

To summarize, the main paradigm shift between DML and Contrastive SSL arises from a few key changes, namely using data-augmentation instead of sampling to obtain the positive/negative pairs, the use of deeper networks, and the use of a predictor network, which we note in Figure 4 . One of the most prominent methods coming from the paradigm shift to SSL in the deep learning family is SimCLR.

SimCLR learns visual representations by encouraging similarity between two augmented views of an image. In SimCLR the two views are formed by applying a combination of transformations including random resizing, cropping, color jittering, and random blurring. After encoding each view, SimCLR uses a projector , often a MLP (multi-layer perceptron) followed by a ReLU (rectified linear unit) activation, to map the initial embeddings into another space where the constrative loss if applied to encourage similarity between the views. For downstream tasks, extracting the representation before the projector has been shown to improve performance. Further discussions of the role of the projector are in sections 2.6.1 and 3.2 .

Another key ingredient along with the InfoNCE loss used in SimCLR is the non-parametric softmax introduced by Wu et al. (2018) . This name is motivated by removing the need to have a "parametrized" linear layer on top of the representation to compute the softmax by instead comparing representations with each others. This loss formulation already contained a temperature parameter in the softmax which is responsible for increasing or decreasing the sharpness of events in predictions. Other noteworthy developements include Schroff et al. (2015) use triplet loss with active triplet selection (hard positive, hard negative) either online from the current mini-batch or from a past checkpoint akin to momentum networks (discussed in section 2.3 ). Weinberger and Saul (2009) introduced push-pull weighting, to push negatives apart while pulling positives together, in a triplet loss to increase the margin of K-NN based models. Tian et al. (2020a) introduced the possibility of many positive views.

Aside from forming positives using semantic preserving transformations, mining positive pairs naturally arising in data is also possible. An iconic triplet loss is based on video frames where the positive pairs come from nearby frames (while negatives are from far away frames) developed in Sermanet et al. (2018) coined Time-Contrastive (TC) Time-Contrastive Learning. Nonlinear ICA ( Hyvarinen and Morioka, 2016 ) introduced a proof that you can learn the log PDF when doing classification tasks. Alexey et al. (2015) trains a classification pretext task by transforming image patches in comparison to different transformations of image patches. One disadvantage is that this setup can involve too many classes leading performance to degrade on downstream tasks. To overcome this, NCE has been successfully employed in Mnih and Teh (2012) ; Mnih and Kavukcuoglu (2013) to modify the denominator in order not to loop over all classes. This is an alternative to sampling based estimation of the gradient that was found to be less stable ( Bengio and Senécal, 2003 ; Bengio and Senécal, 2008 ) . This introduces the concept of will become momentum encoder by imposing that features maps do not vary quickly referred to as proximal algorithm ( Parikh et al., 2014 ) . One other consideration in SSL motivated by the DML is the idea of “hard negative data mining” where the negative samples are intentionaly selected to be close to but distinct from the positives to form a more challenging learning objective. Next we describe an alternative to deep metric learning based on self-distillation.

### 2.3 The Self-Distillation Family: BYOL/SimSIAM/DINO

Self-distillation methods such as BYOL ( Grill et al., 2020 ) , SimSIAM ( Chen and He, 2021 ) , DINO ( Caron et al., 2021 ) , along with their variants rely on a simple mechanism: feeding two different views to two encoders, and mapping one to the other by means of a predictor. To prevent the encoders from collapsing by predicting a constant for any input, various techniques are employed. A common approach to prevent collapse is to update one of the two encoder weights with a running average of the other encoder’s weights. We discuss the particularities of each method.

BYOL (bootstrap your own latent) first introduced self-distillation as a means to avoid collapse. BYOL uses two networks along with a predictor to map the outputs of one network to the other. The network predicting the output is called the online or student network while the network producing the target is called the target or teacher network. Each network receives a different view of the same image formed by image transformations including random resizing, cropping, color jittering, and brightness alterations. The student network is updated throughout training using gradient descent. The teacher network is updated with an exponential moving average (EMA) updates of the weights of the online network. The slow updates induced by exponential moving average creates an asymmetry that is crucial to BYOL’s success. The loss can be defined as ℒ BYOL ​ ( θ s , γ ) = 𝔼 ( 𝒙 , t 1 , t 2 ) ∼ ( X , T 1 , T 2 ) ​ [ ‖ renorm ⁡ ( p γ ​ ( f θ s ​ ( t 1 ​ ( 𝒙 ) ) ) ) − renorm ⁡ ( f θ t ​ ( t 2 ​ ( 𝒙 ) ) ) ‖ 2 2 ] \mathcal{L}_{\rm BYOL}\left(\theta_{\rm s},\gamma\right)=\mathbb{E}_{({\bm{x}},t_{1},t_{2})\sim(X,T_{1},T_{2})}\left[\left\|\mathrm{renorm}(p_{\gamma}(f_{\theta_{\rm s}}(t_{1}({\bm{x}}))))-\mathrm{renorm}(f_{\theta_{\rm t}}(t_{2}({\bm{x}})))\right\|_{2}^{2}\right] (9) where the two vectors in representation space are automatically ℓ 2 \ell_{2} -normalized i.e. renorm ⁡ ( 𝒗 ) = 𝒗 max ⁡ ( ‖ 𝒗 ‖ 2 + ϵ ) , \displaystyle\mathrm{renorm}({\bm{v}})=\frac{{\bm{v}}}{\max(\|{\bm{v}}\|_{2}+{\epsilon})}, (10) where ϵ \epsilon is often set at 1 − 12 1^{-12} . f θ s f_{\theta_{\rm s}} is the online encoder network often denoted as the student parametrized by θ s \theta_{\rm s} , and p γ p_{\gamma} is the predictor network parameterized by γ \gamma . 𝒙 ∼ X {\bm{x}}\sim X is the input sampled from the data distribution X X , and t 1 ​ ( 𝒙 ) , t 2 ​ ( 𝒙 ) t_{1}({\bm{x}}),t_{2}({\bm{x}}) are two augmented views of 𝒙 {\bm{x}} where t 1 ∼ T 1 , t 2 ∼ T 2 t_{1}\sim T_{1},t_{2}\sim T_{2} are two data augmentations. The target network f θ t f_{\theta_{\rm t}} is of the same architecture as the student and is updated by EMA with ξ \xi controlling to what degree the target network preserves its history as in θ t ← ξ ​ θ t + ( 1 − ξ ) ​ θ s \theta_{\rm t}\leftarrow\xi\theta_{\rm t}+(1-\xi)\theta_{\rm s} with initialization η = θ s \eta=\theta_{\rm s} .

SimSiam is aimed at understanding which components in BYOL are most important. SimSiam showed that the EMA was not necessary in practice, even if it led to a small boost in performance. This enabled the use of a simplified loss defined by ℒ SimSIAM ​ ( θ s , γ ) = 𝔼 ( 𝒙 , t 1 , t 2 ) ​ [ ‖ renorm ⁡ ( p γ ​ ( f θ s ​ ( t 1 ​ ( 𝒙 ) ) ) ) − sg ⁡ ( renorm ⁡ ( f θ s ​ ( t 2 ​ ( 𝒙 ) ) ) ) ‖ 2 2 ] , \mathcal{L}_{\rm SimSIAM}\left(\theta_{\rm s},\gamma\right)=\mathbb{E}_{({\bm{x}},t_{1},t_{2})}\left[\|\mathrm{renorm}(p_{\gamma}(f_{\theta_{\rm s}}(t_{1}({\bm{x}}))))-\mathrm{sg}(\mathrm{renorm}(f_{\theta_{\rm s}}(t_{2}({\bm{x}}))))\|_{2}^{2}\right], (11) where for clarity we omit the distribution over which x , t 1 , t 2 x,t_{1},t_{2} are sampled from. Several works have aimed at understanding how BYOL and SimSiam avoid collapse such as Tian et al. (2021) or Halvagal et al. (2022) , where they found that the asymmetry between the two branches is the key, as well the training dynamics which regularize the variance of the embeddings implicitly.

DINO performs a centering of the output of the student network using a running mean (to avoid sensitivity to mini-batch size) and discretize (smoothly) the representations by means of a softmax with a temperate τ \tau usually taken to be around 0.1 0.1 as in ℒ DINO ​ ( θ s , γ ) = 𝔼 ( 𝒙 , t 1 , t 2 ) ​ [ CrossEnt ⁡ ( softmax ⁡ ( f θ s ​ ( t 1 ​ ( 𝒙 ) ) / τ ) , sg ⁡ ( softmax ⁡ ( center ⁡ ( f θ t ​ ( t 2 ​ ( 𝒙 ) ) ) / τ ) ) ) ] , \mathcal{L}_{\rm DINO}\left(\theta_{\rm s},\gamma\right)=\mathbb{E}_{({\bm{x}},t_{1},t_{2})}\left[\mathrm{CrossEnt}\left(\mathrm{softmax}(f_{\theta_{\rm s}}(t_{1}({\bm{x}}))/\tau),\mathrm{sg}(\mathrm{softmax}(\mathrm{center}(f_{\theta_{\rm t}}(t_{2}({\bm{x}})))/\tau))\right)\right], (12) where akin to BYOL the teacher again has a moving average of the student network’s weights, usually with value ξ \xi following a cosine schedule from 0.996 0.996 to 1 1 during training. The discretization in DINO caused by the softmax can be interepreted as an online clustering mechanism, where the last layer before the softmax contains the clustering prototypes and its weight. As such, the output of the penultimate layer is clustered using the weights of the last layer.

iBOT builds on DINO and combines its objective with a masked image modeling objective applied in latent space directly. Here, the target reconstruction is not the image pixels but the same patches embedded through the teacher network.

DINOv2 further builds on iBOT and improves its performance significantly in both linear and k-NN evaluations by improving the training recipe, the architecture, and by introducing additional regularizers such as KoLeo ( Sablayrolles et al., 2018 ) . In addition, DINOv2 curates a larger pretraining dataset consisting of 142 million images (further discussion in Table 1 ).

Many other methods belong to this self-distillation family. MoCo is another popular method based on building a dictionary look-up that was shown to in some cases to surpass supervised learning on segmentation and object detection benchmarks He et al. (2020a) . Originally the momentum encoder was introduced as a substitute for a queue in contrastive learning ( He et al., 2020a ) , which extends the result of ( Dosovitskiy et al., 2014 ) . MoCo’s moving average uses a relatively large momentum with a default value of ξ = 0.999 \xi=0.999 . This higher momentum value works much better than a smaller value of say ξ = 0.9 \xi=0.9 . When SimCLR introduced the use of a projector and stronger data-augmentations, MoCoV2 ( Chen et al., 2020d ) followed suite with stronger data-augmentations and a projector head to boost performance. In a similar spirit, ISD ( Tejankar et al., 2021 ) compares a query distribution to anchors from the student distribution using KL-divergence that relaxes the binary distinction between positive and negative samples. MSF ( Koohpayegani et al., 2021 ) compares a query’s nearest neighbor representation to the student target’s representation and then minimize the ℓ 2 \ell_{2} distnace between them with renormalization (akin to cosine similarity maximization). Another approach, SSCD builds on the contrastive objective to the task of copy detection outperforming copy detection models and other contrastive methods ( Pizzi et al., 2022 ) . Aside from the widespread use of the contrastive objective, many more methods employ similar running average updates as part of their training mechanism. For example, self-distillation ( Hinton et al., 2015 ; Furlanello et al., 2018 ) , Deep Q Network in reinforcement learning ( Mnih et al., 2013 ) , Mean Teacher in semi-supervised learning ( Tarvainen and Valpola, 2017 ) , and even model average in supervised and generative modeling ( Jean et al., 2014 ) .

### 2.4 The Canonical Correlation Analysis Family: VICReg/BarlowTwins/SWAV/W-MSE

The SSL canonical correlation analysis family originates with the Canonical Correlation Framework (CCA) ( Hotelling, 1992 ) . The high-level goal of CCA is to infer the relationship between two variables by analyzing their cross-covariance matrices. Specifically, let 𝑿 ∈ ℝ D {\bm{X}}\in\mathbb{R}^{D} and 𝒀 ∈ ℝ D {\bm{Y}}\in\mathbb{R}^{D} . The CCA framework seeks two transformations 𝑼 = f x ​ ( 𝑿 ) {\bm{U}}=f_{x}({\bm{X}}) and 𝑽 = f y ​ ( 𝒀 ) {\bm{V}}=f_{y}({\bm{Y}}) such that ℒ = − ∑ n = 1 N ⟨ 𝑼 n , 𝑽 n ⟩ , \displaystyle\mathcal{L}=-\sum_{n=1}^{N}\langle{\bm{U}}_{n},{\bm{V}}_{n}\rangle, such that ​ 1 N ​ ∑ n = 1 N 𝑼 n = 1 N ​ ∑ n = 1 N 𝑽 n = 𝟎 ⏟ zero-mean representations , 1 N ​ 𝑼 T ​ 𝑼 = 1 N ​ 𝑽 T ​ 𝑽 = 𝑰 ⏟ identity covariance representations , \displaystyle\text{ such that }\underbrace{\frac{1}{N}\sum_{n=1}^{N}{\bm{U}}_{n}=\frac{1}{N}\sum_{n=1}^{N}{\bm{V}}_{n}=\mathbf{0}}_{\text{zero-mean representations}},\underbrace{\frac{1}{N}{\bm{U}}^{T}{\bm{U}}=\frac{1}{N}{\bm{V}}^{T}{\bm{V}}={\bm{I}}}_{\text{identity covariance representations}}, (13) with d d (the dimension of the output mappings) such that d ≤ min ⁡ ( dim ( 𝑿 ) , dim ( 𝒀 ) ) d\leq\min(\dim({\bm{X}}),\dim({\bm{Y}})) . Linear CCA ( Hotelling, 1992 ) considers the two mappings to be linear in which case the optimal parameters can be found through the SVD of Σ x − 1 2 ​ Σ x ​ y ​ Σ y − 1 2 \Sigma_{x}^{-\frac{1}{2}}\Sigma_{xy}\Sigma_{y}^{-\frac{1}{2}} , involving the covariance matrices of 𝑿 , 𝒀 {\bm{X}},{\bm{Y}} and their cross-covariance. A major advance in the study nonlinear CCA was achieved by Breiman and Friedman (1985) in the univariate output setting, and by Makur et al. (2015) in the multivariate output setting, by connecting the solution to eq. 13 to the Alternating Conditional Expectation (ACE) method. Painsky et al. (2020) study the link between the optimal representation for nonlinear CCA using the Alternating Conditional Expectation proving new theoretical bounds that lead to further refinements of CCA.