Emerging Properties in Self-Supervised Vision Transformers

arxiv:2104.14294

Original source · Paper JSON

Self-supervised learning and vision · 2021-04-29

Cites (1) · Cited by (13)

Complete cleaned Markdown · Source Markdown including bibliography

Read in parts

1. Part 1
2. Part 2
3. Part 3
4. Part 4
5. Part 5
6. Part 6

Contents

• Emerging Properties in Self-Supervised Vision Transformers
• Abstract
• 1 Introduction
• 2 Related work
• Self-supervised learning.
• Self-training and knowledge distillation.
• 3 Approach
• 3.1 SSL with Knowledge Distillation
• Teacher network.
• Network architecture.
• Avoiding collapse.
• 3.2 Implementation and evaluation protocols
• Vision Transformer.
• Implementation details.
• Evaluation protocols.
• 4 Main Results
• 4.1 Comparing with SSL frameworks on ImageNet
• Comparing with the same architecture.
• Comparing across architectures.
• 4.2 Properties of ViT trained with SSL
• 4.2.1 Nearest neighbor retrieval with DINO ViT
• Image Retrieval.
• Copy detection.
• 4.2.2 Discovering the semantic layout of scenes
• Video instance segmentation.
• Probing the self-attention map.
• 4.2.3 Transfer learning on downstream tasks
• 5 Ablation Study of DINO
• 5.1 Importance of the Different Components
• Importance of the patch size.
• 5.2 Impact of the choice of Teacher Network
• Building different teachers from the student.
• Analyzing the training dynamic.
• 5.3 Avoiding collapse
• 5.4 Compute requirements
• 5.5 Training with small batches
• 6 Conclusion
• Acknowledgement.
• Appendix
• A Additional Results
• k k -NN classification.
• Self-supervised ImageNet pretraining of ViT.
• Low-shot learning on ImageNet.
• B Methodology Comparison
• Relation to MoCo-v2 and BYOL.
• Relation to SwAV.
• Details on the Softmax(batch) variant.
• Validating our implementation.
• Relation to other works.
• C Projection Head
• BN-free system.
• L2-normalization bottleneck in projection head.
• Output dimension.
• GELU activations.
• D Additional Ablations
• Online centering.
• Sharpening.
• Longer training.
• The teacher outperforms the student.
• Self-attention maps from supervised versus self-supervised learning.
• Impact of the number of heads in ViT-S.
• E Multi-crop
• Range of scales in multi-crop.
• Multi-crop in different self-supervised frameworks.
• Training BYOL with multi-crop.
• F Evaluation Protocols
• F.1 k k -NN classification
• F.2 Linear classification
• ViT-S representations for linear eval.
• ViT-B representations for linear eval.
• G Self-Attention Visualizations
• H Class Representation
• Supplementary Material for Self-Supervised Learning of Visual Transformers with a Momentum Teacher
• Appendix
• A Additional Results
• k k -NN classification.
• Self-supervised ImageNet pretraining of ViT.
• Low-shot learning on ImageNet.
• B Methodology Comparison
• Relation to MoCo-v2 and BYOL.
• Relation to SwAV.
• Details on the Softmax(batch) variant.
• Validating our implementation.
• Relation to other works.
• C Projection Head
• BN-free system.
• L2-normalization bottleneck in projection head.
• Output dimension.
• GELU activations.
• D Additional Ablations
• Online centering.
• Sharpening.
• Longer training.
• The teacher outperforms the student.
• Self-attention maps from supervised versus self-supervised learning.
• Impact of the number of heads in ViT-S.
• E Multi-crop
• Range of scales in multi-crop.
• Multi-crop in different self-supervised frameworks.
• Training BYOL with multi-crop.
• F Evaluation Protocols
• F.1 k k -NN classification
• F.2 Linear classification
• ViT-S representations for linear eval.
• ViT-B representations for linear eval.
• G Self-Attention Visualizations
• H Class Representation