Emerging Properties in Self-Supervised Vision Transformers — part 5/6

Cleaned-text code points [61205, 76351). Bibliography omitted here; source.md includes it.

We evaluate the features obtained with DINO applied on ViT-S on low-shot learning. In Tab. 3 , we report the validation accuracy of a logistic regression trained on frozen features ( frozen ) with 1% and 10% labels. The logistic regression is trained with the cyanure library [ 16 ] . When comparing models with a similar number of parameters and image/sec, we observe that our features are on par with state-of-the-art semi-supervised models. Interestingly, this performance is obtained by training a multi-class logistic regression on frozen features, without data augmentation nor finetuning .

### B Methodology Comparison

We compare the performance of different self-supervised frameworks, MoCo-v2 [ 6 ] , SwAV [ 3 ] and BYOL [ 12 ] when using convnet or ViT. In Tab. 4 , we see that when trained with ResNet-50 (convnet), DINO performs on par with SwAV and BYOL. However, DINO unravels its potential with ViT, outperforming MoCo-v2, SwAV and BYOL by large margins (+4.3% with linear and +6.2% with k-NN evaluations). In the rest of this section, we perform ablations to better understand the performance of DINO applied to ViT. In particular, we provide a detailed comparison with methods that either use a momentum encoder, namely MoCo-v2 and BYOL, and methods that use multi-crop, namely SwAV.

##### Relation to MoCo-v2 and BYOL.

In Tab. 5 , we present the impact of ablating components that differ between DINO, MoCo-v2 and BYOL: the choice of loss, the predictor in the student head, the centering operation, the batch normalization in the projection heads, and finally, the multi-crop augmentation. The loss in DINO is a cross-entropy on sharpened softmax outputs ( CE ) while MoCo-v2 uses the InfoNCE contrastive loss ( INCE ) and BYOL a mean squared error on l2-normalized outputs ( MSE ). No sharpening is applied with the MSE criterion. Though, DINO surprisingly still works when changing the loss function to MSE , but this significantly alters the performance (see rows ( \rownumber 1, \rownumber 2) and ( \rownumber 4, \rownumber 9)). We also observe that adding a predictor has little impact ( \rownumber 1, \rownumber 3). However, in the case of BYOL, the predictor is critical to prevent collapse ( \rownumber 7, \rownumber 8) which is consistent with previous studies [ 7 , 12 ] . Interestingly, we observe that the teacher output centering avoids collapse without predictor nor batch normalizations in BYOL ( \rownumber 7, \rownumber 9), though with a significant performance drop which can likely be explained by the fact that our centering operator is designed to work in combination with sharpening. Finally, we observe that multi-crop works particularly well with DINO and MoCo-v2, removing it hurts performance by 2 − 4 % 2-4\% ( \rownumber 1 versus \rownumber 4 and, \rownumber 5 versus \rownumber 6). Adding multi-crop to BYOL does not work out-of-the-box ( \rownumber 7, \rownumber 10) as detailed in Appendix E and further adaptation may be required.

##### Relation to SwAV.

In Tab. 6 , we evaluate the differences between DINO and SwAV: the presence of the momentum encoder and the operation on top of the teacher output. In absence of the momentum, a copy of the student with a stop-gradient is used. We consider three operations on the teacher output: Centering , Sinkhorn-Knopp or a Softmax along the batch axis. The Softmax is similar to a single Sinkhorn-Knopp iteration as detailed in the next paragraph. First, these ablations show that using a momentum encoder significantly improves the performance for ViT ( \rownumber 3 versus \rownumber 6, and \rownumber 2 versus \rownumber 5). Second, the momentum encoder also avoids collapse when using only centering (row \rownumber 1). In the absence of momentum, centering the outputs does not work ( \rownumber 4) and more advanced operations are required ( \rownumber 5, \rownumber 6). Overall, these ablations highlight the importance of the momentum encoder, not only for performance but also to stabilize training, removing the need for normalization beyond centering.

##### Details on the Softmax(batch) variant.

The iterative Sinkhorn-Knopp algorithm [ 8 ] used in SwAV [ 3 ] is implemented simply with the following PyTorch style code. ⬇

When performing a single Sinkhorn iteration ( num_iters=1 ) the implementation can be highly simplified into only two lines of code, which is our softmax(batch) variant:

We have seen in Tab. 6 that this highly simplified variant of SwAV works competitively with SwAV. Intuitively, the softmax operation on the batch axis allows to select for each dimension (or “cluster”) its best matches in the batch.

##### Validating our implementation.

We observe in Tab. 4 that our reproduction of BYOL, MoCo-v2, SwAV matches or outperforms the corresponding published numbers with ResNet-50. Indeed, we obtain 72.7 % 72.7\% for BYOL while [ 12 ] report 72.5 % 72.5\% in this 300 300 -epochs setting. We obtain 71.1 % 71.1\% for MoCo after 300 300 epochs of training while [ 6 ] report 71.1 % 71.1\% after 800 800 epochs of training. Our improvement compared to the implementation of [ 6 ] can be explained by the use of a larger projection head (3-layer, use of batch-normalizations and projection dimension of 256 256 ).

##### Relation to other works.

DINO is also related to UIC [ 14 ] that use outputs from the previous epoch as hard pseudo-labels for “unsupervised classification”. However, we use centering to prevent collapse while UIC resorts to balance sampling techniques as in [ 8 ] . Our work can be interpreted as a soft UIC variant with momentum teacher.

The concurrent work CsMI [ 77 ] also exhibits strong performance with simple k-NN classifiers on ImageNet, even with convnets. As DINO, CsMI combines a momentum network and multi-crop training, which we have seen are both crucial for good k-NN performance in our experiments with ViTs. We believe studying this work would help us identifying more precisely the components important for good k k -NN performance and leave this investigation for future work.

### C Projection Head

Similarly to other self-supervised frameworks, using a projection head [ 4 ] improves greatly the accuracy of our method. The projection head starts with a n n -layer multi-layer perceptron (MLP). The hidden layers are 2048d and are with gaussian error linear units (GELU) activations. The last layer of the MLP is without GELU. Then we apply a ℓ 2 \ell_{2} normalization and a weight normalized fully connected layer [ 7 , 22 ] with K K dimensions. This design is inspired from the projection head with a “prototype layer” used in SwAV [ 3 ] . We do not apply batch normalizations.

##### BN-free system.

Unlike standard convnets, ViT architectures do not use batch normalizations (BN) by default.

Therefore, when applying DINO to ViT we do not use any BN also in the projection heads. In this table we evaluate the impact of adding BN in the heads. We observe that adding BN in the projection heads has little impact, showing that BN is not important in our framework. Overall, when applying DINO to ViT, we do not use any BN anywhere, making the system entirely BN-free. This is a great advantage of DINO + ViT to work at state-of-the-art performance without requiring any BN. Indeed, training with BN typically slows down trainings considerably, especially when these BN modules need to be synchronized across processes [ 13 , 3 , 2 , 12 ] .

##### L2-normalization bottleneck in projection head.

We illustrate the design of the projection head with or without l2-normalization bottleneck in Fig. 2 .

We evaluate the accuracy of DINO models trained with or without l2-normalization bottleneck and we vary the number of linear layers in the projection head. With l2 bottleneck, the total number of linear layers is n + 1 n+1 ( n n from the MLP and 1 1 from the weight normalized layer) while without bottleneck the total number of linear layers is n n in the head. In this table, we report ImageNet top-1 k k -NN evaluation accuracy after 100 epochs pre-training with ViT-S/16. The output dimensionality K K is set to 4096 4096 in this experiment. We observe that DINO training fails without the l2-normalization bottleneck when increasing the depth of the projection head. L2-normalization bottleneck stabilizes the training of DINO with deep projection head. We observe that increasing the depth of the projection head improves accuracy. Our default is to use a total of 4 linear layers: 3 are in the MLP and one is after the l2 bottleneck.

##### Output dimension.

In this table, we evaluate the effect of varying the output dimensionality K K .

We observe that a large output dimensionality improves the performance. We note that the use of l2-normalization bottleneck permits to use a large output dimension with a moderate increase in the total number of parameters. Our default is to use K K equals to 65536 and d = 256 d=256 for the bottleneck.

##### GELU activations.

By default, the activations used in ViT are gaussian error linear units (GELU).

Therefore, for consistency within the architecture, we choose to use GELU also in the projection head. We evaluate the effect of using ReLU instead of GELU in this table and observe that changing the activation unit to ReLU has relatively little impact.

### D Additional Ablations

We have detailed in the main paper that the combination of centering and sharpening is important to avoid collapse in DINO. We ablate the hyperparameters for these two operations in the following. We also study the impact of training length and some design choices for the ViT networks.

##### Online centering.

We study the impact of the smoothing parameters in the update rule for the center c c used in the output of the teacher network.

The convergence is robust to a wide range of smoothing, and the model only collapses when the update is too slow, i.e., m = 0.999 m=0.999 .

##### Sharpening.

We enforce sharp targets by tuning the teacher softmax temperature parameter τ t \tau_{t} . In this table, we observe that a temperature lower than 0.06 0.06 is required to avoid collapse.

When the temperature is higher than 0.06 0.06 , the training loss consistently converges to l ​ n ​ ( K ) ln(K) . However, we have observed that using higher temperature than 0.06 0.06 does not collapse if we start the training from a smaller value and increase it during the first epochs. In practice, we use a linear warm-up for τ t \tau_{t} from 0.04 0.04 to 0.07 0.07 during the first 30 30 epochs of training. Finally, note that τ → 0 \tau\rightarrow 0 (extreme sharpening) correspond to the argmax operation and leads to one-hot hard distributions.

##### Longer training.

We observe in this table that longer training improves the performance of DINO applied to ViT-Small.

This observation is consistent with self-supervised results obtained with convolutional architectures [ 4 ] . We note that in our experiments with BYOL on ViT-S, training longer than 300 300 epochs has been leading to worse performance compare our 300 300 epochs run. For this reason we report BYOL for 300 epochs in Tab. while SwAV, MoCo-v2 and DINO are trained for 800 epochs.

##### The teacher outperforms the student.

We have shown in Fig. that the momentum teacher outperforms the student with ViT and we show in this Figure that it is also the case with ResNet-50.

The fact that the teacher continually outperforms the student further encourages the interpretation of DINO as a form of Mean Teacher [ 24 ] self-distillation. Indeed, as motivated in Tarvainen et al. [ 24 ] , weight averaging usually produces a better model than the individual models from each iteration [ 20 ] . By aiming a target obtained with a teacher better than the student, the student’s representations improve. Consequently, the teacher also improves since it is built directly from the student weights.

##### Self-attention maps from supervised versus self-supervised learning.

We evaluate the masks obtained by thresholding the self-attention maps to keep 80% of the mass.

We compare the Jaccard similarity between the ground truth and these masks on the validation images of PASCAL VOC12 dataset for different ViT-S trained with different frameworks. The properties that self-attention maps from ViT explicitly contain the scene layout and, in particular, object boundaries is observed across different self-supervised methods.

##### Impact of the number of heads in ViT-S.

We study the impact of the number of heads in ViT-S on the accuracy and throughput (images processed per second at inference time on a singe V100 GPU).

We find that increasing the number of heads improves the performance, at the cost of a slighlty worse throughput. In our paper, all experiments are run with the default model DeiT-S [ 25 ] , i.e. with 6 6 heads only.

### E Multi-crop

In this Appendix, we study a core component of DINO: multi-crop training [ 3 ] .

##### Range of scales in multi-crop.

For generating the different views, we use the RandomResizedCrop method from torchvision.transforms module in PyTorch.

We sample two global views with scale range ( s , 1 ) (s,1) before resizing them to 224 2 224^{2} and 6 6 local views with scale sampled in the range ( 0.05 , s ) (0.05,s) resized to 96 2 96^{2} pixels. Note that we arbitrarily choose to have non-overlapping scaling range for the global and local views following the original design of SwAV. However, the ranges could definitely be overlapping and experimenting with finer hyperparameters search could lead to a more optimal setting. In this table, we vary the parameter s s that controls the range of scales used in multi-crop and find the optimum to be around 0.3 0.3 in our experiments. We note that this is higher than the parameter used in SwAV which is of 0.14 0.14 .

##### Multi-crop in different self-supervised frameworks.

We compare different recent self-supervised learning frameworks, namely MoCo-v2 [ 6 ] , BYOL [ 12 ] and SwAV [ 3 ] with ViT-S/16 architecture.

For fair comparisons, all models are pretrained either with two 224 2 224^{2} crops or with multi-crop [ 3 ] training, i.e. two 224 2 224^{2} crops and six 96 2 96^{2} crops for each image. We report k k -NN and linear probing evaluations after 300 epochs of training. Multi-crop does not benefit all frameworks equally, which has been ignored in benchmarks considering only the two crops setting [ 7 ] . The effectiveness of multi-crop depends on the considered framework, which positions multi-crop as a core component of a model and not a simple “add-ons” that will boost any framework the same way. Without multi-crop, DINO has better accuracy than other frameworks, though by a moderate margin (1%). Remarkably, DINO benefits the most from multi-crop training ( + 3.4 % +3.4\% in linear eval). Interestingly, we also observe that the ranking of the frameworks depends on the evaluation protocol considered.

##### Training BYOL with multi-crop.

When applying multi-crop to BYOL with ViT-S, we observe the transfer performance is higher than the baseline without multi-crop for the first training epochs.