DINOv2: Learning Robust Visual Features without Supervision — part 1/4

Cleaned-text code points [0, 15817). Bibliography omitted here; source.md includes it.

# DINOv2: Learning Robust Visual Features without Supervision

###### Abstract

The recent breakthroughs in natural language processing for model pretraining on large quantities of data have opened the way for similar foundation models in computer vision. These models could greatly simplify the use of images in any system by producing general-purpose visual features, i.e., features that work across image distributions and tasks without finetuning. This work shows that existing pretraining methods, especially self-supervised methods, can produce such features if trained on enough curated data from diverse sources. We revisit existing approaches and combine different techniques to scale our pretraining in terms of data and model size. Most of the technical contributions aim at accelerating and stabilizing the training at scale. In terms of data, we propose an automatic pipeline to build a dedicated, diverse, and curated image dataset instead of uncurated data, as typically done in the self-supervised literature. In terms of models, we train a ViT model ( Dosovitskiy et al., 2021 ) with 1B parameters and distill it into a series of smaller models that surpass the best available general-purpose features, OpenCLIP ( Ilharco et al., 2021 ) on most of the benchmarks at image and pixel levels.

Reviewed on OpenReview: https://openreview.net/forum?id=a68SUt6zFt

## 1 Introduction

Learning task-agnostic pretrained representations have become the standard in Natural Language Processing (NLP) ( Radford et al., 2019 ; Raffel et al., 2020 ; Chowdhery et al., 2022 ; Hoffmann et al., 2022 ; Touvron et al., 2023 ) . One can use these features “as they are”, i.e., without fine-tuning, and achieve performances on downstream tasks that are significantly better than those produced by task-specific models ( Brown et al., 2020 ) . This success has been fueled by pretraining on large quantities of raw text using pretext objectives, such as language modeling ( Radford et al., 2017 ) or word vectors ( Devlin et al., 2019 ) , that require no supervision.

Following this paradigm shift in NLP, we expect similar “foundation” models to appear in computer vision ( Bommasani et al., 2021 ) . These models should generate visual features that work out of the box on any task, both at the image level, e.g., image classification, and pixel level, e.g., segmentation. Most promising efforts towards these foundation models focus on text-guided pretraining, i.e., using a form of textual supervision to guide the training of the features ( Joulin et al., 2016 ; Mahajan et al., 2018 ; Radford et al., 2021 ) . This form of text-guided pretraining limits the information that can be retained about the image since captions only approximate the rich information in images, and complex pixel-level information may not surface with this supervision. Furthermore, these image encoders require aligned text-image corpora and hence, do not offer the flexibility of their text counterparts, that is, to learn from raw data alone.

An alternative to text-guided pretraining is self-supervised learning ( Caron et al., 2018 ; Chen et al., 2020 ; He et al., 2022 ) where features are learned from images alone. These approaches are conceptually closer to pretext tasks such as language modeling and can capture information at the image and pixel level ( Caron et al., 2021 ) . Additionally, the features output by self-supervised models have been shown to exhibit various useful properties, and have enabled enabled a wide variety of applications ( Amir et al., 2022 ; Tumanyan et al., 2022 ; Ofri-Amar et al., 2023 ; Hamilton et al., 2022 ) . However, despite their potential to learn general-purpose features, most of the advances in self-supervised learning were made in the context of pretraining on a small curated dataset, ImageNet-1k ( Russakovsky et al., 2015 ) . Some efforts on scaling these approaches beyond ImageNet-1k have been attempted ( Caron et al., 2019 ; Goyal et al., 2021 ; Goyal et al., 2022a ) , but they focused on uncurated datasets, which typically lead to a significant drop in the quality of the features. This is explained by the lack of control over the data quality and diversity, which are essential to produce good features.

In this work, we explore if self-supervised learning has the potential to learn general-purpose visual features if pretrained on a large quantity of curated data. We revisit existing discriminative self-supervised approaches that learn features at both the image and patch level, such as iBOT ( Zhou et al., 2022a ) , and we reconsider some of their design choices under the lens of a larger dataset. Most of our technical contributions are tailored toward stabilizing and accelerating discriminative self-supervised learning when scaling in model and data sizes. These improvements make our approach around 2 × \times faster and require 3 × \times less memory than similar discriminative self-supervised methods, allowing us to leverage longer training with larger batch sizes.

Regarding pretraining data, we have built an automatic pipeline to filter and rebalance datasets from an extensive collection of uncurated images. This pipeline is inspired by pipelines used in NLP ( Wenzek et al., 2020 ) , where data similarities are used instead of external metadata and do not require manual annotation. A major difficulty when dealing with images in the wild is to rebalance concepts and avoid overfitting on a few dominant modes. In this work, a naive clustering approach works reasonably well to resolve this issue. We gathered a small but diverse corpus of 142M images to validate our approach.

Finally, we provide a variety of pretrained visual models, called DINOv2, trained with different Vision Transformers (ViT) ( Dosovitskiy et al., 2016 ) architectures on our data. We release all the models and the code to retrain DINOv2 on any data. We validate the quality of DINOv2 on various computer vision benchmarks at both image and pixel levels as we scale them, as summarized in Fig. 2 . We conclude that self-supervised pretraining alone is a good candidate for learning transferable frozen features that are competitive with the best openly available weakly-supervised models.

## 2 Related Work

##### Intra-image self-supervised training.

A first family of self-supervised methods focuses on pretext tasks built from the image, i.e., extracting a signal from the image to be predicted from the rest of the image. This idea has become prevalent with the work of Doersch et al. (2015) , where they train by predicting the context of a given patch. Many other pretext tasks were introduced based on, for example, re-colorizing images ( Zhang et al., 2016 ) , predicting transformations ( Gidaris et al., 2018 ) , inpainting ( Pathak et al., 2016 ) or patch re-ordering ( Noroozi & Favaro, 2016 ; Misra & Maaten, 2020 ) . Recently, the emergence of patch-based architectures, like ViTs, has led to a revisit of inpainting for pre-training ( He et al., 2022 ; Bao et al., 2021 ; El-Nouby et al., 2021 ) , potentially in feature space ( Assran et al., 2023 ; Baevski et al., 2022 ) . Of particular interest, He et al. (2022) show that a masked auto-encoder (MAE) learns features that provide substantial improvements when finetuned on downstream tasks. This property of MAEs has been further validated on video ( Tong et al., 2022 ) , audio ( Xu et al., 2022 ) , and across other modalities ( Girdhar et al., 2023 ) . However, their features require supervised finetuning, while our features perform well out of the box.

##### Discriminative self-supervised learning.

The second line of work, closer to ours, is using discriminative signals between images or groups of images to learn features. This family of methods has roots in early deep learning work ( Hadsell et al., 2006 ) but became popular with the emergence of instance classification methods ( Dosovitskiy et al., 2016 ; Bojanowski & Joulin, 2017 ; Wu et al., 2018 ) . Several improvements were made based either on instance-level objectives ( Hénaff et al., 2019 ; He et al., 2020 ; Chen & He, 2021 ; Chen et al., 2020 ; Grill et al., 2020 ; Caron et al., 2021 ) or clustering ( Caron et al., 2018 ; Asano et al., 2020 ; Caron et al., 2020 ) . These methods provide performant frozen features on standard benchmarks like ImageNet ( Russakovsky et al., 2015 ) , but they are hard to scale to larger model sizes ( Chen et al., 2021 ) . In this work, we revisit the training of these approaches in the context of large pretraining datasets and models. In particular, we build on top of Zhou et al. (2022a) that we find particularly suited for scaling.

##### Scaling self-supervised pretraining.

A growing body of work has focused on the scaling abilities of self-supervised learning in terms of data and model size ( Caron et al., 2019 ; Goyal et al., 2019 ; Tian et al., 2021 ; Goyal et al., 2022a ) . Most of these works use large quantities of uncurated data to train models without supervision. They show evidence that discriminative methods scale with data, but because of the poor quality of the pretraining data, most of the results are obtained by finetuning the features. Of particular interest, Goyal et al. (2021) have also shown that these methods benefit from scaling in model size given enough pretrained data. This line of work questions the ability of self-supervised methods to work on any data while we focus on producing the best pretrained encoders.

##### Automatic data curation.

Our dataset construction borrows from the image retrieval community ( Weinzaepfel et al., 2021 ; Radenović et al., 2018b ; Berman et al., 2019 ; Douze et al., 2009 ; Tolias et al., 2016 ; Revaud et al., 2019 ) . In particular, the use of retrieval to augment the training set has been studied in the context of semi-supervised learning ( Yalniz et al., 2019 ) . Similarly, others have used hashtags or other metadata ( Mahajan et al., 2018 ; Radford et al., 2021 ) or pretrained vision encoders ( Schuhmann et al., 2021 ; Schuhmann et al., 2022 ) to filter uncurated datasets. Unlike these works, we use no pretrained encoders, metadata nor supervision to filter images and leverage visual similarity between images. Our approach is inspired by text curation pipelines ( Wenzek et al., 2020 ) , where a language model is trained on Wikipedia to score texts extracted from an uncurated source.

## 3 Data Processing

We assemble our curated LVD-142M dataset by retrieving, from a large pool of uncurated data, images that are close to those in several curated datasets. We describe below the main components in our data pipeline including the curated/uncurated data sources, the image deduplication step and the retrieval system. Our pipeline does not require any metadata or text and directly works with images, as shown in Fig. 3 . We refer the reader to appendix A for more details on our approach.

Data sources. Our selection of curated datasets is detailed in the appendix (Table 15 ) and contains ImageNet-22k, the train split of ImageNet-1k, Google Landmarks and several fine-grained datasets. For the uncurated data source, we collect a raw unfiltered dataset of images from a publicly available repository of crawled web data. From each web page in the repository, we extract URL links of images from <img> tags. We discard URLs that are unsafe or restricted by domains, and post-process the downloaded images (PCA hash deduplication, NSFW filtering, and blurring identifiable faces). This results in 1.2B unique images.

Deduplication. We apply the copy detection pipeline of Pizzi et al. (2022) to the uncurated data and remove near-duplicate images. This reduces redundancy and increases diversity among images. We also remove near-duplicates of images contained in the test or validation set of any benchmark used in this work.

Self-supervised image retrieval. We build our curated pretraining dataset by retrieving images from our uncurated data source that are close to images in our curated sources. In order to do this, we first compute an image embedding using a self-supervised ViT-H/16 network pretrained on ImageNet-22k, and use cosine-similarity as a distance measure between images. Then, we perform k-means clustering of the uncurated data. Given a query dataset for retrieval, if it is large enough we retrieve N N (typically 4) nearest neighbors for each query image. If it is small, we sample M M images from the cluster corresponding to each query image. Although visual inspection seemed to indicate good retrieval quality for N N much larger than 4, this leads to more collisions (images that are nearest-neighbor retrievals of multiple queries). We choose N = 4 N=4 as it provides a good tradeoff in that sense.

Implementation Details. The deduplication and retrieval stages of our pipeline rely on the Faiss library ( Johnson et al., 2019 ) to efficiently index and compute batch searches of nearest embeddings. In particular, we heavily leverage its support for GPU-accelerated indices, using inverted file indices with product quantization codes ( Jegou et al., 2010 ) . The whole processing is distributed on a compute cluster of 20 nodes equipped with 8 V100-32GB GPUs and takes less than two days to produce the LVD-142M dataset.

## 4 Discriminative Self-supervised Pre-training

We learn our features with a discriminative self-supervised method that can be seen as a combination of DINO and iBOT losses with the centering of SwAV ( Caron et al., 2020 ) . We also add a regularizer to spread features and a short high-resolution training phase. We rapidly introduce each of these approaches, but more details can be found in the related papers, or in our open-sourced code.

• Image-level objective ( Caron et al., 2021 ) . We consider the cross-entropy loss between the features extracted from a student and a teacher network. Both features are coming from the class token of a ViT, obtained from different crops of the same image. We pass the student class token through the student DINO head. This head is an MLP model outputting a vector of scores, that we call "prototype scores". We then apply a softmax to obtain p s p_{s} . Similarly, we apply the teacher DINO head to the teacher class token to obtain teacher prototype scores. We then apply a softmax followed by a centering with moving average (or a Sinkhorn-Knopp centering as detailed thereafter) to obtain p t p_{t} . The DINO loss term corresponds to: ℒ D ​ I ​ N ​ O = − ∑ p t log p s {\mathcal{L}}_{DINO}=-\sum p_{t}\log p_{s}

We learn the parameters of the student and build the teacher head with an exponential moving average of past iterates ( He et al., 2020 ) .

• Patch-level objective ( Zhou et al., 2022a ) . We randomly mask some of the input patches given to the student, but not to the teacher. We then apply the student iBOT head to the student mask tokens. Similarly, we apply the teacher iBOT head to the (visible) teacher patch tokens corresponding to the ones masked in the student. We then apply the softmax and centering steps as above, and obtain the iBOT loss term: ℒ i ​ B ​ O ​ T = − ∑ i p t ​ i log p s ​ i {\mathcal{L}}_{iBOT}=-\sum_{i}p_{ti}\log p_{si} , where i i are patch indices for masked tokens. Similarly to above, we learn the parameters of the student, and build the teacher head through exponential moving average.

• Untying head weights between both objectives. Both the DINO and the iBOT loss use a learnable MLP projection head. It is applied to the output tokens and the loss is compute atop. In Zhou et al. (2022a) , an ablation study shows that sharing parameters between the DINO and iBOT heads leads to better performance. At scale, we observed that the opposite is true, and we therefore use two separate heads in all our experiments.