Emerging Properties in Self-Supervised Vision Transformers — part 6/6
Cleaned-text code points [76351, 81260). Bibliography omitted here; source.md includes it.
However, the transfer performance growth rate is slowing down and declines after a certain amount of training. We have performed learning rate, weight decay, multi-crop parameters sweeps for this setting and systematically observe the same pattern. More precisely, we experiment with { 1 e − 5 1e^{-5} , 3 e − 5 3e^{-5} , 1 e − 4 1e^{-4} , 3 e − 4 3e^{-4} , 1 e − 3 1e^{-3} , 3 e − 3 3e^{-3} } for learning rate base values, with { 0.02 0.02 , 0.05 0.05 , 0.1 0.1 } for weight decay and with different number of small crops: {2, 4, 6}. All our runs are performed with synchronized batch normalizations in the heads. When using a low learning rate, we did not observe the performance break point, i.e. the transfer performance was improving continually during training, but the overall accuracy was low. We have tried a run with multi-crop training on ResNet-50 where we also observe the same behavior. Since integrating multi-crop training to BYOL is not the focus of this study we did not push that direction further. However, we believe this is worth investigating why multi-crop does not combine well with BYOL in our experiments and leave this for future work.
### F Evaluation Protocols
#### F.1 k k -NN classification
Following the setting of Wu et al . [ 27 ] , we evaluate the quality of features with a simple weighted k k Nearest Neighbor classifier. We freeze the pretrained model to compute and store the features of the training data of the downstream task. To classify a test image x x , we compute its representation and compare it against all stored training features T T . The representation of an image is given by the output [CLS] token: it has dimensionality d = 384 d=384 for ViT-S and d = 768 d=768 for ViT-B. The top k k NN (denoted 𝒩 k \mathcal{N}_{k} ) are used to make a prediction via weighted voting. Specifically, the class c c gets a total weight of ∑ i ∈ 𝒩 k α i 𝟏 c i = c \sum_{i\in\mathcal{N}_{k}}\alpha_{i}\mathbf{1}_{c_{i}=c} , where α i \alpha_{i} is a contribution weight. We use α i = exp ( T i x / τ ) \alpha_{i}=\exp(T_{i}x/\tau) with τ \tau equals to 0.07 0.07 as in [ 27 ] which we do not tune. We evaluate different values for k k and find that k = 20 k=20 is consistently leading to the best accuracy across our runs. This evaluation protocol does not require hyperparameter tuning, nor data augmentation and can be run with only one pass over the downstream dataset.
#### F.2 Linear classification
Following common practice in self-supervised learning, we evaluate the representation quality with a linear classifier. The projection head is removed, and we train a supervised linear classifier on top of frozen features. This linear classifier is trained with SGD and a batch size of 1024 1024 during 100 100 epochs on ImageNet. We do not apply weight decay. For each model, we sweep the learning rate value. During training, we apply only random resizes crops (with default parameters from PyTorch RandomResizedCrop ) and horizontal flips as data augmentation. We report central-crop top-1 accuracy. When evaluating convnets, the common practice is to perform global average pooling on the final feature map before the linear classifier. In the following, we describe how we adapt this design when evaluating ViTs.
##### ViT-S representations for linear eval.
Following the feature-based evaluations in BERT [ 9 ] , we concatenate the [CLS] tokens from the l l last layers.
We experiment with the concatenation of a different number l l of layers and similarly to [ 9 ] we find l = 4 l=4 to be optimal.
##### ViT-B representations for linear eval.
With ViT-B we did not find that concatenating the representations from the last l l layers to provide any performance gain, and consider the final layer only ( l = 1 l=1 ).
In this setting, we adapt the pipeline used in convnets with global average pooling on the output patch tokens. We concatenate these pooled features to the final [CLS] output token.
### G Self-Attention Visualizations
We provide more self-attention visualizations in Fig. 1 and in Fig. 3 . The images are randomly selected from COCO validation set, and are not used during training of DINO. In Fig. 1 , we show the self-attention from the last layer of a DINO ViT-S/8 for several reference points.
### H Class Representation
As a final visualization, we propose to look at the distribution of ImageNet concepts in the feature space from DINO. We represent each ImageNet class with the average feature vector for its validation images. We reduce the dimension of these features to 30 with PCA, and run t-SNE with a perplexity of 20, a learning rate of 200 for 5000 iterations. We present the resulting class embeddings in Fig. 4 . Our model recovers structures between classes: similar animal species are grouped together, forming coherent clusters of birds (top) or dogs, and especially terriers (far right).