DINOv2: Learning Robust Visual Features without Supervision — part 3/4

Cleaned-text code points [31679, 47527). Bibliography omitted here; source.md includes it.

We also want to validate that our features are competitive with state-of-the-art open-source weakly supervised models. To this end, we compare on ImageNet-1k, using the linear evaluation, to three off-the-shelf methods with several architectural variants. For all models, we run the linear evaluation using our code, after making sure that our numbers match those reported in technical reports and papers. We show the result of this evaluation in Table 4 . We see that our backbone, surpases the performance of OpenCLIP with a ViT-G/14 architecture ( + 0.3 % +0.3\% ) and EVA-CLIP with a ViT-g/14 ( + 0.1 % +0.1\% ). At the same time, we also observe that our performance on the ImageNet-V2 test set is significantly better ( + 1.1 % +1.1\% versus EVA-CLIP), indicating better generalization. For the remainder of this section, we report OpenCLIP-G as a reference for weakly-supervised models.

##### Can we finetune the encoders?

We question if the ability of our models to produce high quality frozen features impact their performance when finetuned with supervision on a specific dataset. While this is not core to this paper, this experiment is indicative of whether we have involuntarily specialized our models to the setting of linear evaluations of frozen features. To run this sanity check, we apply the finetuning pipeline from Touvron et al. (2022) , without tweaking hyper-parameters. In Table 5 , we show that the Top-1 accuracy on the validation set of ImageNet-1k improves by more than + 2 % +2\% when the backbone is finetuned. This is true both when using models at resolution 224 224 and 448 448 . Further gains can be obtained by tuning the hyper-parameters of the finetuning, but this is beyond the goal of this sanity check. Nonetheless, our best finetuned performance ( 88.9 % 88.9\% ) is only a couple of percent below ( − 2.2 % -2.2\% ) the absolute state of the arts ( 91.1 % 91.1\% ), obtained by Chen et al. (2023a) . As DINOv2 leads to features that are strong in both the linear and finetuning settings, a strong property of our approach is that finetuning is optional .

##### Robustness analysis.

To complement our study, and probe the generalization of our features, we evaluate our ImageNet-1k models trained with linear classification heads on domain generalization benchmarks. We use the best performing linear classifier as described above and simply run inference on those benchmarks. Please note that most results in the literature are obtained with models that are finetuned end-to-end on ImageNet-1k. We show the result of this experiment in Table 6 . When comparing with state-of-the-art SSL methods, our models shows drastically better robustness ( + 29.6 % +29.6\% on A ( Hendrycks et al., 2021b ) , + 22.1 % +22.1\% on R ( Hendrycks et al., 2021a ) and + 23.0 % +23.0\% on Sketch ( Wang et al., 2019 ) compared to iBOT). Our model also improves upon the best weakly-supervised model on ImageNet-A while lagging behind on R and Sketch.

### 7.2 Additional Image and Video classification Benchmarks

In this section we study the generalization of our features on downstream classification benchmarks. We consider two sets of evaluations in that context. On one hand, we use large and finegrained datasets such as iNaturalist and Places205. On the other, we use the 12 image classification tasks originally proposed in SimCLR ( Chen et al., 2020 ) . For iNaturalist 2018, iNaturalist 2021, and Places205, we train a linear classifier with data augmentations as in Sec. 7.1 We report top-1 accuracy for those three datasets in Table 7 . Interestingly, our model significantly outperforms OpenCLIP ViT-G/14 on both variants of iNaturalist ( + 8.6 % +8.6\% and + 9.7 % +9.7\% for 2018 and 2021 respectively), and lags slightly behind on Places 205 ( − 2.3 % -2.3\% ).

In a second set of evaluations, we measure the performance of our model on video action recognition even though our features were not trained on videos.. We evaluated features on three datasets, namely UCF-101 ( Soomro et al., 2012 ) , Kinetics-400 ( Kay et al., 2017 ) and Something-Something v2 ( Goyal et al., 2017 ) . For this evaluation, we pick 8 8 evenly spaced frames in the video and train a linear classifier on the average of the features for UCF and K-400. For SSv2, we opt for concatenation to retain more temporal information than with feature averaging. For each dataset, we measure average accuracy and report the results in Table 7 . We see that amongst self-supervised approaches, our model clearly sets a new state of the art. Moreover, our model matches the accuracy of the OpenCLIP features on UCF and Kinetics ( + 0.1 % +0.1\% and + 0.5 % +0.5\% respectively) and clearly outperforms them on SSv2 ( + 2.5 % +2.5\% ). This is particularly interesting, as SSv2 requires a much richer understanding of the video frames.

Finally, in Table 8 , we compare selected frozen features on 12 transfer classification benchmarks initially proposed by Chen et al. (2020) . This benchmark covers scenes, objects (food, cars, planes), and textures. We replace the Birdsnap dataset with CUB because the former was not publicly available in its entirety. We follow the experimental protocol as outlined by Chen et al. (2020) , namely training a logistic regression on precomputed features. Our model significantly outperforms state-of-the-art SSL models, with most notable differences on Stanford Cars ( + 14.8 % +14.8\% versus DINO ViT-B/8) and FGVC Aircraft ( + 14.8 % +14.8\% versus iBOT ViT-L/16). Even though these benchmarks favor text-guided pretraining, our features are still competitive with OpenCLIP on most classification benchmarks, with the exception of a few datasets, especially SUN ( − 5.3 % -5.3\% ) and Cars ( − 4.7 % -4.7\% ).

### 7.3 Instance Recognition

In this experiment, we probe our model on the task of instance-level recognition using a non-parametric approach. Images from a database are ranked according to their cosine similarity with a query image. We evaluated our model and compare to baselines on Paris and Oxford, that are landmark recognition benchmarks. We also evaluated on Met, a dataset of artworks from the Metropolitan museum, and AmsterTime, containing street view images matched to archival images of Amsterdam. We measure performance by computing the mean average precision and report our results in Table 9 . We see that our features significantly outperform both SSL ( + 41 % +41\% mAP on Oxford-Hard), and weakly-supervised ( + 34 % +34\% mAP on Oxford-Hard) ones. It is interesting to see that our features perform well across task granularities, both at the category-level and instance-level. This is a desirable property for strong off-the-shelf computer vision features.

### 7.4 Dense Recognition Tasks

We probe the quality of patch-level features extracted from our network on several dense downstream tasks. We consider semantic image segmentation and monocular depth estimation in several settings and we conduct evaluations on multiple datasets for each.

##### Semantic segmentation.

For our semantic segmentation evaluation, we consider two different setups. Linear : a linear layer is trained to predict class logits from a patch tokens. It is used to produce a low-resolution logit map (eg 32x32 for a model with patch size 16), which is then upsampled to full resolution (512x512) to obtain a segmentation map. This procedure is extremely simple but cannot easily produce high-resolution segmentations. +ms : a boosted version of the linear setup. We concatenate the patch tokens of the 4 last layers, use a larger image resolution of 640, and use multiscale test-time augmentations to improve the predictions. We report the performance of our model variants as well as the baselines on three datasets under the two setups in Table 10 .

Our models show very good performance on all datasets and for all setups. Interestingly, our evaluation using +ms is on par with fully finetuning MAE with an Upernet decoder ( 53.0 53.0 versus 53.6 53.6 mIoU). This is surprising because we use a significantly simpler predictor. Also, our best model, when evaluated using the boosted recipe, almost matches the state of the art on Pascal VOC ( 86.2 86.2 versus 89.0 89.0 mIoU).

##### Frozen backbone in a SOTA pipeline.

In a final experiment, we freeze our backbone, and plug it into a ViT-Adapter Chen et al. (2023b) with a Mask2former head ( Cheng et al., 2022 ) . We tune the weights of the adapter and head, but keep the backbone frozen, meaning 66% of the weights are frozen. This allows for a lighter segmentation training than full end-to-end fine-tuning. With this setup, we reach 60.2 60.2 mIoU on ADE20k, close to the competitive state of the art, standing at 62.9 mIoU ( Wang et al., 2022 ) . Although our setup for this experiment doesn’t makes use of the optimisations described in Sec. 5 , the segmentation training in this experiment took 28 hours on 16 V100 GPUs.

##### Depth estimation.

In this experiment, we evaluate our patch-level features on three monocular depth estimation benchmarks: NYUd, KITTI and zero-shot transfer from NYUd to SUN3d. We follow the evaluation protocol of Li et al. (2022b) . We consider three different setups for this evaluation. lin. 1 : we extract the last layer of the frozen transformer and concatenate the [CLS] token to each patch token. Then we bi-linearly upsample the tokens by a factor of 4 to increase the resolution. Finally we train a simple linear layer using a classification loss by dividing the depth prediction range in 256 uniformly distributed bins and use a linear normalization following Bhat et al. (2021) . lin. 4 : we use the same protocol that we use with one layer, but concatenate the tokens from layers l = { 3 , 6 , 9 , 12 } l=\{3,6,9,12\} for ViT-S/B, l = { 5 , 12 , 18 , 24 } l=\{5,12,18,24\} for ViT-L, and l = { 10 , 20 , 30 , 40 } l=\{10,20,30,40\} for ViT-g. DPT : we use the DPT decoder ( Ranftl et al., 2021 ) on top of our frozen models and setup a regression task. We scale the size of the head following the dimension of the features for each architecture. We show results for all baselines, all datasets and all setups in Table 11 .

From this table, we see that our features clearly surpass the best SSL and WSL features available. It is interesting to see that iBOT features extracted from a ViT-L outperform the ones of OpenCLIP with a ViT-G. This observation supports the intuition that caption-based feature learning fails to learn subtle patterns like this one. Also, our model, with the DPT decoder and frozen backbone, matches or exceeds the performance of the recent work of Li et al. (2022b) . Finally, the out-of-domain generalization result on SUN-RGBd shows that our features allow very good transfer between domains. A depth prediction module trained on indoor scenes from NYUd generalizes pretty well to the outdoor examples of SUN-RGBd.

### 7.5 Qualitative Results

In this final section of the empirical evaluation of our features, we propose a few qualitative analyses.

##### Semantic Segmentation and Depth Estimation.

We show some qualitative results for our dense prediction evaluations: segmentation on ADE20K in Fig. 7 and depth estimation on NYUd, KITTI and SUN RGB-D in Fig. 7 . We compare DINOv2 with OpenCLIP with a linear classifier on each dataset. While not perfect, the linear segmentation model using our DINOv2 backbone produces good results and behaves much better than the OpenCLIP one under this evaluation setup. Indeed, the segmentation mask produced by OpenCLIP-G shows many artifacts and disconnected components. The qualitative results on depth estimation clearly illustrate the quantitative gap between OpenCLIP and DINOv2. These results highlight that our features, as well as the features extracted from OpenCLIP, are able to linearly separate complex information such as depth, even though neither was trained with this type of information. However, our features lead to a much smoother depth estimation, with less artifacts. Some objects such as the chair on the SUN RGB-D image are completely ignored by OpenCLIP and correctly positioned using our features.

##### Out-of-distribution generalization.

We show a few examples of applying the depth prediction and segmentation linear classifiers to out-of-distribution examples in Fig. 8 . The qualitative results support our claim that our features transfer between domains. The quality of the depth and segmentation predicted for pictures of animals, or paintings is very good, even though the domains are very different.

##### PCA of patch features.

We show the results of the principal component analysis (PCA) performed on the patch features extracted by our model. We keep only patches with a positive value after we threshold the first component. This procedure turns out to separate the image’s main object from the background. We compute a second PCA on the remaining patches across three images depicting the same category. We color the three first components with three different colors and present the results in Fig. 1 and 9 . There are two interesting observations: first, our unsupervised foreground / background detector, based on detecting the highest variance direction, performs very well and is capable of delineating the boundary of the main object in the picture. Second, the other components correspond to "parts" of objects and match well for images of the same category. This is an emerging property – our model was not trained to parse parts of objects.

##### Patch matching.

Finally, we explore what type of information our patch-level features contain by matching them across images. We start by detecting the foreground object using the procedure described above. Then, we compute the euclidean distance between patch features extracted from two images and map them by solving an assignment problem. In order to reduce the number of matches, we then apply a non-maximum suppression to keep only the salient ones. In Fig. 10 , we show some examples of such matchings.

We observe that the features seem to capture information about semantic regions that serve similar purpose in different objects or animals. For instance, the wing of a plane matches the wing of a bird. We also observe that the model is robust to style (image versus drawing), and to large variation of poses (see the elephant).

## 8 Fairness and Bias Analysis

We conduct two fairness evaluations of our models. We probe for geographical fairness and potential harmful label associations. For both evaluations, we experiment with our largest ViT-g model.

### 8.1 Geographical Fairness

We evaluate geographical fairness on the Dollar Street dataset introduced in De Vries et al. (2019) using the evaluation protocol of Goyal et al. (2022b) . This benchmark compares performance across countries and income levels. It contains 16,073 images from 289 households across 54 countries. The task is to recognize 94 concepts that vary visually between households based on income or location. In Table 12 , we compare our model with SEERv2 ( Goyal et al., 2022a ) , a model trained on a geographically diverse set of images. Our model is slightly fairer across regions and incomes than the SEERv2 model and significantly better than the supervised baseline reported by Goyal et al. (2022a) . However, we still observe a significant difference between regions, particularly in Africa, where our model performance drops by 25.7% compared to Europe. This shows that our model is still biased toward Western countries. Similarly, our model performs significantly better on high-income households than low-income ones, with a difference of 31.7%. Despite improvements, we observe significant biases in our models toward wealthy households from Western countries.

### 8.2 Gender, Skintones and Age