Emerging Properties in Self-Supervised Vision Transformers — part 4/6
Cleaned-text code points [45498, 61205). Bibliography omitted here; source.md includes it.
Therefore, when applying DINO to ViT we do not use any BN also in the projection heads. In this table we evaluate the impact of adding BN in the heads. We observe that adding BN in the projection heads has little impact, showing that BN is not important in our framework. Overall, when applying DINO to ViT, we do not use any BN anywhere, making the system entirely BN-free. This is a great advantage of DINO + ViT to work at state-of-the-art performance without requiring any BN. Indeed, training with BN typically slows down trainings considerably, especially when these BN modules need to be synchronized across processes [ 13 , 3 , 2 , 12 ] .
##### L2-normalization bottleneck in projection head.
We illustrate the design of the projection head with or without l2-normalization bottleneck in Fig. 9 .
We evaluate the accuracy of DINO models trained with or without l2-normalization bottleneck and we vary the number of linear layers in the projection head. With l2 bottleneck, the total number of linear layers is n + 1 n+1 ( n n from the MLP and 1 1 from the weight normalized layer) while without bottleneck the total number of linear layers is n n in the head. In this table, we report ImageNet top-1 k k -NN evaluation accuracy after 100 epochs pre-training with ViT-S/16. The output dimensionality K K is set to 4096 4096 in this experiment. We observe that DINO training fails without the l2-normalization bottleneck when increasing the depth of the projection head. L2-normalization bottleneck stabilizes the training of DINO with deep projection head. We observe that increasing the depth of the projection head improves accuracy. Our default is to use a total of 4 linear layers: 3 are in the MLP and one is after the l2 bottleneck.
##### Output dimension.
In this table, we evaluate the effect of varying the output dimensionality K K .
We observe that a large output dimensionality improves the performance. We note that the use of l2-normalization bottleneck permits to use a large output dimension with a moderate increase in the total number of parameters. Our default is to use K K equals to 65536 and d = 256 d=256 for the bottleneck.
##### GELU activations.
By default, the activations used in ViT are gaussian error linear units (GELU).
Therefore, for consistency within the architecture, we choose to use GELU also in the projection head. We evaluate the effect of using ReLU instead of GELU in this table and observe that changing the activation unit to ReLU has relatively little impact.
### D Additional Ablations
We have detailed in the main paper that the combination of centering and sharpening is important to avoid collapse in DINO. We ablate the hyperparameters for these two operations in the following. We also study the impact of training length and some design choices for the ViT networks.
##### Online centering.
We study the impact of the smoothing parameters in the update rule for the center c c used in the output of the teacher network.
The convergence is robust to a wide range of smoothing, and the model only collapses when the update is too slow, i.e., m = 0.999 m=0.999 .
##### Sharpening.
We enforce sharp targets by tuning the teacher softmax temperature parameter τ t \tau_{t} . In this table, we observe that a temperature lower than 0.06 0.06 is required to avoid collapse.
When the temperature is higher than 0.06 0.06 , the training loss consistently converges to l n ( K ) ln(K) . However, we have observed that using higher temperature than 0.06 0.06 does not collapse if we start the training from a smaller value and increase it during the first epochs. In practice, we use a linear warm-up for τ t \tau_{t} from 0.04 0.04 to 0.07 0.07 during the first 30 30 epochs of training. Finally, note that τ → 0 \tau\rightarrow 0 (extreme sharpening) correspond to the argmax operation and leads to one-hot hard distributions.
##### Longer training.
We observe in this table that longer training improves the performance of DINO applied to ViT-Small.
This observation is consistent with self-supervised results obtained with convolutional architectures [ 4 ] . We note that in our experiments with BYOL on ViT-S, training longer than 300 300 epochs has been leading to worse performance compare our 300 300 epochs run. For this reason we report BYOL for 300 epochs in Tab. 2 while SwAV, MoCo-v2 and DINO are trained for 800 epochs.
##### The teacher outperforms the student.
We have shown in Fig. 6 that the momentum teacher outperforms the student with ViT and we show in this Figure that it is also the case with ResNet-50.
The fact that the teacher continually outperforms the student further encourages the interpretation of DINO as a form of Mean Teacher [ 24 ] self-distillation. Indeed, as motivated in Tarvainen et al. [ 24 ] , weight averaging usually produces a better model than the individual models from each iteration [ 20 ] . By aiming a target obtained with a teacher better than the student, the student’s representations improve. Consequently, the teacher also improves since it is built directly from the student weights.
##### Self-attention maps from supervised versus self-supervised learning.
We evaluate the masks obtained by thresholding the self-attention maps to keep 80% of the mass.
We compare the Jaccard similarity between the ground truth and these masks on the validation images of PASCAL VOC12 dataset for different ViT-S trained with different frameworks. The properties that self-attention maps from ViT explicitly contain the scene layout and, in particular, object boundaries is observed across different self-supervised methods.
##### Impact of the number of heads in ViT-S.
We study the impact of the number of heads in ViT-S on the accuracy and throughput (images processed per second at inference time on a singe V100 GPU).
We find that increasing the number of heads improves the performance, at the cost of a slighlty worse throughput. In our paper, all experiments are run with the default model DeiT-S [ 25 ] , i.e. with 6 6 heads only.
### E Multi-crop
In this Appendix, we study a core component of DINO: multi-crop training [ 3 ] .
##### Range of scales in multi-crop.
For generating the different views, we use the RandomResizedCrop method from torchvision.transforms module in PyTorch.
We sample two global views with scale range ( s , 1 ) (s,1) before resizing them to 224 2 224^{2} and 6 6 local views with scale sampled in the range ( 0.05 , s ) (0.05,s) resized to 96 2 96^{2} pixels. Note that we arbitrarily choose to have non-overlapping scaling range for the global and local views following the original design of SwAV. However, the ranges could definitely be overlapping and experimenting with finer hyperparameters search could lead to a more optimal setting. In this table, we vary the parameter s s that controls the range of scales used in multi-crop and find the optimum to be around 0.3 0.3 in our experiments. We note that this is higher than the parameter used in SwAV which is of 0.14 0.14 .
##### Multi-crop in different self-supervised frameworks.
We compare different recent self-supervised learning frameworks, namely MoCo-v2 [ 6 ] , BYOL [ 12 ] and SwAV [ 3 ] with ViT-S/16 architecture.
For fair comparisons, all models are pretrained either with two 224 2 224^{2} crops or with multi-crop [ 3 ] training, i.e. two 224 2 224^{2} crops and six 96 2 96^{2} crops for each image. We report k k -NN and linear probing evaluations after 300 epochs of training. Multi-crop does not benefit all frameworks equally, which has been ignored in benchmarks considering only the two crops setting [ 7 ] . The effectiveness of multi-crop depends on the considered framework, which positions multi-crop as a core component of a model and not a simple “add-ons” that will boost any framework the same way. Without multi-crop, DINO has better accuracy than other frameworks, though by a moderate margin (1%). Remarkably, DINO benefits the most from multi-crop training ( + 3.4 % +3.4\% in linear eval). Interestingly, we also observe that the ranking of the frameworks depends on the evaluation protocol considered.
##### Training BYOL with multi-crop.
When applying multi-crop to BYOL with ViT-S, we observe the transfer performance is higher than the baseline without multi-crop for the first training epochs.
However, the transfer performance growth rate is slowing down and declines after a certain amount of training. We have performed learning rate, weight decay, multi-crop parameters sweeps for this setting and systematically observe the same pattern. More precisely, we experiment with { 1 e − 5 1e^{-5} , 3 e − 5 3e^{-5} , 1 e − 4 1e^{-4} , 3 e − 4 3e^{-4} , 1 e − 3 1e^{-3} , 3 e − 3 3e^{-3} } for learning rate base values, with { 0.02 0.02 , 0.05 0.05 , 0.1 0.1 } for weight decay and with different number of small crops: {2, 4, 6}. All our runs are performed with synchronized batch normalizations in the heads. When using a low learning rate, we did not observe the performance break point, i.e. the transfer performance was improving continually during training, but the overall accuracy was low. We have tried a run with multi-crop training on ResNet-50 where we also observe the same behavior. Since integrating multi-crop training to BYOL is not the focus of this study we did not push that direction further. However, we believe this is worth investigating why multi-crop does not combine well with BYOL in our experiments and leave this for future work.
### F Evaluation Protocols
#### F.1 k k -NN classification
Following the setting of Wu et al . [ 27 ] , we evaluate the quality of features with a simple weighted k k Nearest Neighbor classifier. We freeze the pretrained model to compute and store the features of the training data of the downstream task. To classify a test image x x , we compute its representation and compare it against all stored training features T T . The representation of an image is given by the output [CLS] token: it has dimensionality d = 384 d=384 for ViT-S and d = 768 d=768 for ViT-B. The top k k NN (denoted 𝒩 k \mathcal{N}_{k} ) are used to make a prediction via weighted voting. Specifically, the class c c gets a total weight of ∑ i ∈ 𝒩 k α i 𝟏 c i = c \sum_{i\in\mathcal{N}_{k}}\alpha_{i}\mathbf{1}_{c_{i}=c} , where α i \alpha_{i} is a contribution weight. We use α i = exp ( T i x / τ ) \alpha_{i}=\exp(T_{i}x/\tau) with τ \tau equals to 0.07 0.07 as in [ 27 ] which we do not tune. We evaluate different values for k k and find that k = 20 k=20 is consistently leading to the best accuracy across our runs. This evaluation protocol does not require hyperparameter tuning, nor data augmentation and can be run with only one pass over the downstream dataset.
#### F.2 Linear classification
Following common practice in self-supervised learning, we evaluate the representation quality with a linear classifier. The projection head is removed, and we train a supervised linear classifier on top of frozen features. This linear classifier is trained with SGD and a batch size of 1024 1024 during 100 100 epochs on ImageNet. We do not apply weight decay. For each model, we sweep the learning rate value. During training, we apply only random resizes crops (with default parameters from PyTorch RandomResizedCrop ) and horizontal flips as data augmentation. We report central-crop top-1 accuracy. When evaluating convnets, the common practice is to perform global average pooling on the final feature map before the linear classifier. In the following, we describe how we adapt this design when evaluating ViTs.
##### ViT-S representations for linear eval.
Following the feature-based evaluations in BERT [ 9 ] , we concatenate the [CLS] tokens from the l l last layers.
We experiment with the concatenation of a different number l l of layers and similarly to [ 9 ] we find l = 4 l=4 to be optimal.
##### ViT-B representations for linear eval.
With ViT-B we did not find that concatenating the representations from the last l l layers to provide any performance gain, and consider the final layer only ( l = 1 l=1 ).
In this setting, we adapt the pipeline used in convnets with global average pooling on the output patch tokens. We concatenate these pooled features to the final [CLS] output token.
### G Self-Attention Visualizations
We provide more self-attention visualizations in Fig. 8 and in Fig. 10 . The images are randomly selected from COCO validation set, and are not used during training of DINO. In Fig. 8 , we show the self-attention from the last layer of a DINO ViT-S/8 for several reference points.
### H Class Representation
As a final visualization, we propose to look at the distribution of ImageNet concepts in the feature space from DINO. We represent each ImageNet class with the average feature vector for its validation images. We reduce the dimension of these features to 30 with PCA, and run t-SNE with a perplexity of 20, a learning rate of 200 for 5000 iterations. We present the resulting class embeddings in Fig. 11 . Our model recovers structures between classes: similar animal species are grouped together, forming coherent clusters of birds (top) or dogs, and especially terriers (far right).
## Supplementary Material for Self-Supervised Learning of Visual Transformers with a Momentum Teacher
Anonymous ICCV submission Paper ID 7530
## Appendix
### A Additional Results
##### k k -NN classification.
In Tab. 1 , we evaluate the frozen representations given by ResNet-50 or ViT-small pre-trained with DINO with two evaluation protocols: linear or k k -NN. For both evaluations, we extract representations from a pre-trained network without using any data augmentation. Then, we perform classification either with weighted k k -NN or with a linear regression learned with cyanure library [ 16 ] . In Tab. 1 we see that ViT-S accuracies are better than accuracies obtained with RN50 both with a linear or a k k -NN classifier. However, the performance gap when using the k k -NN evaluation is much more significant than when considering linear evaluation. For example on ImageNet 1%, ViT-S outperforms ResNet-50 by a large margin of + 14.1 % +14.1\% with k k -NN evaluation. This suggests that transformers architectures trained with DINO might offer more model flexibility that benefits the k k -NN evaluation. K K -NN classifiers have the great advantage of being fast and light to deploy, without requiring any domain adaptation. Overall, ViT trained with DINO provides features that combine particularly well with k k -NN classifiers.
##### Self-supervised ImageNet pretraining of ViT.
In this experiment, we study the impact of pretraining a supervised ViT model with our method. In Tab. 2 , we compare the performance of supervised ViT models that are initialized with different pretraining or guided during training with an additional pretrained convnet. The first set of models are pretrained with and without supervision on the large curated dataset composed of 300M images. The second set of models are trained with hard knowledge distillation from a pretrained supervised RegNetY [ 56 ] . The last set of models do not use any additional data nor models, and are initialized either randomly or after a pretraining with DINO on ImageNet. Compare to random initialization, pretraining with DINO leads to a performance gain of +1%. This is not caused by a longer training since pretraining with supervision instead of DINO does not improve performance. Using self-supervised pretraining reduces the gap with models pretrained on extra data or distilled from a convnet.
##### Low-shot learning on ImageNet.