A Cookbook of Self-Supervised Learning

arxiv:2304.12210

Original source · Paper JSON

Self-supervised learning and vision · 2023-04-24

Cites (7) · Cited by (1)

Complete cleaned Markdown · Source Markdown including bibliography

Read in parts

1. Part 1
2. Part 2
3. Part 3
4. Part 4
5. Part 5
6. Part 6
7. Part 7
8. Part 8

Contents

• A Cookbook of Self-Supervised Learning
• 1 What is Self-Supervised Learning and Why Bother?
• 1.1 Why a Cookbook for Self-Supervised Learning?
• 2 The Families and Origins of SSL
• 2.1 Origins of SSL
• 2.2 The Deep Metric Learning Family: SimCLR/NNCLR/MeanSHIFT/SCL
• 2.3 The Self-Distillation Family: BYOL/SimSIAM/DINO
• 2.4 The Canonical Correlation Analysis Family: VICReg/BarlowTwins/SWAV/W-MSE
• 2.5 Masked Image Modeling
• 2.6 A Theoretical Unification Of Self-Supervised Learning
• 2.6.1 Theoretical Study of SSL
• Relationship between Contrastive Learning and Other Objectives.
• Unified contrastive losses.
• Hard negative sampling.
• Study of the projector.
• 2.6.2 Dimensional Collapse of Representations
• 2.7 Pretraining Data
• Curated (standard)
• Training with data from the wild
• Weakly-curated training data
• 3 A Cook’s Guide to Successful SSL Training and Deployment
• 3.1 Role of Data-Augmentation
• 3.1.1 Role of multi-crop
• 3.2 Role of the Projector
• Using a projector to handle noisy image augmentations.
• Influence of the projector’s output dimension.
• Influence of the backbone’s output dimension.
• Properties of the representation induced by the projector.
• Training a SSL without a projector.
• 3.3 The Uniform Prior in SSL or the Failure of SSL on Unbalanced Data
• 3.4 Teacher-Student Architecture Specific Tricks
• 3.4.1 Role of the Moving Average Teacher
• 3.4.2 Role of the Predictor in Self-Labeling SSL
• 3.5 Role of Standard Hyper-Parameters
• 3.5.1 Role of Mini-Batch Size
• 3.5.2 Role of Learning Rate (Schedulers) and Optimizers
• 3.5.3 Role of Weight-Decay
• 3.5.4 Vision Transformers Considerations
• 3.6 Techniques for High Performance Masked Image Modeling
• 3.7 Evaluating Your SSL Models
• 3.7.1 Evaluation with labels
• KNN
• Linear
• MLP
• Full Fine-tuning
• 3.7.2 Evaluation without labels
• 3.7.3 Going beyond classification
• 3.7.4 Visual Evaluation
• 3.8 Speeding up Training
• 3.8.1 Distributed Training
• 3.8.2 Even Faster Training with FFCV and Other Speedups
• 3.8.3 Speeding Up Training of Vision Transformers
• 4 Extending Self-Supervised Learning Beyond Images and Classification
• 4.1 Strategies for Other Data Domains
• 4.2 Incorporating Multiple Modalities into SSL Training
• 4.3 Building Feature Extractors with Localization for Dense Prediction Tasks
• 5 Conclusion