DINOv2: Learning Robust Visual Features without Supervision

arxiv:2304.07193

Original source · Paper JSON

Self-supervised learning and vision · 2023-04-14

Cites (6) · Cited by (7)

Complete cleaned Markdown · Source Markdown including bibliography

Read in parts

1. Part 1
2. Part 2
3. Part 3
4. Part 4

Contents

• DINOv2: Learning Robust Visual Features without Supervision
• Abstract
• 1 Introduction
• 2 Related Work
• Intra-image self-supervised training.
• Discriminative self-supervised learning.
• Scaling self-supervised pretraining.
• Automatic data curation.
• 3 Data Processing
• 4 Discriminative Self-supervised Pre-training
• 5 Efficient implementation
• Fast and memory-efficient attention.
• Sequence packing.
• Efficient stochastic depth.
• Fully-Sharded Data Parallel (FSDP).
• Model distillation.
• 6 Ablation Studies
• 6.1 Improved Training Recipe
• 6.2 Pretraining Data Source
• 6.3 Model Size and Data
• 6.4 Loss Components
• 6.5 Impact of Knowledge Distillation
• 6.6 Impact of Resolution
• 7 Results
• 7.1 ImageNet Classification
• How far are we from weakly-supervised models?
• Can we finetune the encoders?
• Robustness analysis.
• 7.2 Additional Image and Video classification Benchmarks
• 7.3 Instance Recognition
• 7.4 Dense Recognition Tasks
• Semantic segmentation.
• Frozen backbone in a SOTA pipeline.
• Depth estimation.
• 7.5 Qualitative Results
• Semantic Segmentation and Depth Estimation.
• Out-of-distribution generalization.
• PCA of patch features.
• Patch matching.
• 8 Fairness and Bias Analysis
• 8.1 Geographical Fairness
• 8.2 Gender, Skintones and Age
• 9 Estimating the Environmental Impact of Training our Models
• Carbon footprint of the whole project.
• 10 Future work and Discussion
• Acknowledgments.
• Appendix A Data Processing
• A.1 Data selection
• A.2 Image similarity
• A.3 Deduplication
• Self-deduplication.
• Relative deduplication
• A.4 Retrieval
• Appendix B Implementation Details
• B.1 Unsupervised pre-training
• KoLeo regularization.
• EMA update for the teacher.
• B.2 High-Resolution adaptation
• B.3 Linear probing evaluation
• Appendix C List of Datasets used
• Instructions for reporting errors