iBOT: Image BERT Pre-Training with Online Tokenizer

arxiv:2111.07832

Original source · Paper JSON

Self-supervised learning and vision · 2021-11-15

Cites (2) · Cited by (5)

Complete cleaned Markdown · Source Markdown including bibliography

Read in parts

1. Part 1
2. Part 2
3. Part 3
4. Part 4

Contents

• iBOT : Image BERT Pre-Training with Online Tokenizer
• Abstract
• 1 Introduction
• 2 Preliminaries
• 2.1 Masked Image Modeling as Knowledge Distillation
• 2.2 Self-Distillation
• 3 iBOT
• 3.1 Framework
• 3.2 Implementation
• Architecture.
• Pre-Training Setup.
• 4 Experiment
• 4.1 Classification on ImageNet-1K
• k k -NN and Linear Probing.
• Fine-Tuning.
• Semi-Supervised and Unsupervised Learning.
• 4.2 Downstream Tasks
• Object Detection and Instance Segmentation on COCO.
• Semantic Segmentation on ADE20K.
• Transfer Learning.
• 4.3 Properties of ViT trained with MIM
• 4.3.1 Discovering the Pattern Layout of Image Patches
• What Patterns Does MIM Learn?
• How Does MIM Help Image Recognition?
• 4.3.2 Discriminative Parts in Self-Attention Map
• 4.3.3 Robustness
• 4.4 Ablation Study on Tokenizer
• 5 Related Work
• Visual Representation Learning.
• Masked Prediction in Images.
• 6 Conclusion
• Appendix A Pseudocode
• Appendix B Multi-Crop
• Stability of MIM Pre-trained with Multi-Crop.
• Stabilizing the Training with Non-Masked Global Crops.
• Range of Scales in Multi-Crop.
• State-of-the-Art Comparison w/o and w/ Multi-Crop.
• Effective Training Epochs.
• Appendix C Additional Implementations
• Fine-Tuning Recipes of Classification on ImageNet-1K.
• Evaluation Protocols of Semi-Supervised Learning on ImageNet-1K.
• Fine-Tuning Recipes of Object Detection and Instance Segmentation on COCO.
• Fine-Tuning Recipes of Semantic Segmentation on ADE20K.
• Part-Wise Linear Probing.
• Appendix D Additional Results
• Object Detection, Instance Segmentation, and Semantic Segmentation.
• k k -NN and Linear Probing with ImageNet-22K.
• Nearest Neighbor Retrieval.
• Robustness against Background Change.
• Robustness against Occlusion.
• Robustness against Shuffle.
• Appendix E Additional Ablations
• Architecture of Projection Head.
• Comparing MIM with Dense Self-Distillation.
• Hard Label versus Soft Label
• Centering and Sharpening.
• Loss Ratio.
• Output Dimension.
• Prediction Ratios.
• Training Epochs.
• Time and Memory Requirements.
• Appendix F Alternative Tokenizers
• Appendix G Visualization
• G.1 Pattern Layout
• Pattern Layout for Patch Tokens.
• Composing Images with Representative Patterns.
• Comparison with Other Methods.
• Pattern Layout for [CLS] Token.
• G.2 Self-Attention Visualizations
• G.3 Sparse Correspondence.
• Instructions for reporting errors