iBOT: Image BERT Pre-Training with Online Tokenizer — part 2/4

Cleaned-text code points [15494, 31218). Bibliography omitted here; source.md includes it.

We by default pre-train iBOT on ImageNet-1K ( Deng et al., 2009 ) training set with AdamW ( Loshchilov & Hutter, 2019 ) optimizer and a batch size of 1024 1024 . We pre-train iBOT with ViT-S/16 for 800 800 epochs, ViT-B/16 for 400 400 epochs, ViT-L/16 for 250 250 epochs, and Swin-T/{7,14} for 300 300 epochs. We also pre-train on ImageNet-22K training set with ViT-B/16 for 80 80 epochs and ViT-L/16 for 50 50 epochs. The learning rate is linearly ramped up during the first 10 10 epochs to its base value scaled with the total batch size: lr = 5 ​ e − 4 × batch ​ _ ​ size / 256 \mathrm{lr}=5e^{-4}\times\mathrm{batch\_size}/256 . We use random MIM, with prediction ratio r r set as 0 0 with a probability of 0.5 0.5 and uniformly sampled from range [ 0.1 0.1 , 0.5 0.5 ] with a probability of 0.5 0.5 . We sum ℒ [CLS] \mathcal{L}_{\texttt{[CLS]}} and ℒ MIM \mathcal{L}_{\mathrm{MIM}} up without scaling.

## 4 Experiment

We first transfer iBOT to downstream tasks, following the standard evaluation protocols adopted in prior arts, the details of which are delayed in Appendix C . We then study several interesting properties of Transformers pre-trained with iBOT. Finally, we give a brief ablation study on the crucial composing of iBOT.

### 4.1 Classification on ImageNet-1K

We consider five classification protocols on ImageNet-1K: k k -NN, linear probing, fine-tuning, semi-supervised learning, and unsupervised learning.

##### k k -NN and Linear Probing.

To evaluate the quality of pre-trained features, we either use a k k -nearest neighbor ( k k -NN) classifier or a linear classifier on the frozen representation. We follow the evaluation protocols in DINO ( Caron et al., 2021 ) . For k k -NN evaluation, we sweep over different numbers of nearest neighbors. For linear evaluation, we sweep over different learning rates. In Tab. 3 , our method reaches a linear probing accuracy 77.9% with ViT-S/16, a linear probing accuracy 79.5% with ViT-B/16, and a k k -NN accuracy 78.0% and linear probing accuracy 81.0% with ViT-L/16, achieving state-of-the-art performance. With Swin-T/{7,14}, iBOT achieves a linear probing accuracy of 78.6% and 79.3% respectively.With ViT-L/16 and ImageNet-22K as pre-training data, iBOT further achieves a linear probing accuracy 82.3% , surpassing previous state of the art, 81.3% with Swin-B/14 by EsViT ( Li et al., 2021a ) . A linear probing accuracy of 79.5% with ViT-B/16 is comparable to 79.8% by SimCLRv2 with RN152 ( 3 × 3\times ) † but with 10 × 10\times less parameters. We underline that the performance gain over DINO gets larger (0.9% w/ ViT-S versus 1.3% w/ ViT-B) with more parameters, suggesting iBOT is more scalable to larger models.

##### Fine-Tuning.

We study the fine-tuning on ImageNet-1K and focus on the comparison with self-supervised methods for Transformers and its supervised baseline ( Rand. ) ( Touvron et al., 2021 ) . As shown in Tab. 3 , iBOT achieves an 82.3%, 84.0%, and 84.8% top-1 accuracy with ViT-S/16, ViT-B/16, and ViT-L/16, respectively. As shown in Tab. 3 , iBOT pre-trained with ImageNet-22K achieves 84.4% and 86.6% top-1 accuracy with ViT-B/16 and ViT-L/16, respectively, outperforming ImageNet-22K pre-trained BEiT by 0.7% and 0.6%. When fine-tuned on an image size of 512, we achieve 87.8% accuracy. We note that, with ViT-L/16, iBOT is 0.4% worse than BEiT using 1K data but 0.6% better using 22K data. This implies that iBOT requires more data to train larger model.

##### Semi-Supervised and Unsupervised Learning.

For semi-supervised learning, we focus our comparison with methods following the unsupervised pre-train, supervised fine-tune paradigm. As shown in Tab. 5 , iBOT advances DINO by 1.6% and 0.8% using 1% and 10% data, respectively, suggesting a higher label efficiency. For unsupervised learning, we use standard evaluation metrics, including accuracy (ACC), adjusted random index (ARI), normalized mutual information (NMI), and Fowlkes-Mallows index (FMI). We compare our methods to SimCLRv2 ( Chen et al., 2020b ) , Self-label ( Asano et al., 2020 ) , InfoMin ( Tian et al., 2020 ) , and SCAN ( Van Gansbeke et al., 2020 ) . As shown in Tab. 5 , we achieve a 32.8% NMI, outperforming the previous state of the art by 1.8%, suggesting MIM helps the model learn stronger visual semantics on a global scale.

### 4.2 Downstream Tasks

##### Object Detection and Instance Segmentation on COCO.

Object detection and instance segmentation require simultaneous object location and classification.We consider Cascade Mask R-CNN ( Cai & Vasconcelos, 2019 ; He et al., 2017 ) that produces bounding boxes and instance masks simultaneously on COCO dataset ( Lin et al., 2014 ) . Several recent works ( Liu et al., 2021b ; Wang et al., 2021a ) proposes Vision Transformers that suit dense downstream tasks. To compare, we include the results of supervised Swin-T ( Liu et al., 2021b ) which shares approximate parameter numbers with ViT-S/16 and its self-supervised counterpart MoBY ( Xie et al., 2021a ) in Tab. 6 . iBOT improves ViT-S’s AP b from 46.2 to 49.4 and AP m from 40.1 to 42.6, surpassing both supervised Swin-T and its self-supervised counterpart by a nontrivial margin. With ViT-B/16, iBOT achieves an AP b of 51.2 and an AP m of 44.2, surpassing previous best results by a large margin.

##### Semantic Segmentation on ADE20K.

Semantic segmentation can be seen as a pixel-level classification problem. We mainly consider two segmentation settings on ADE20K dataset ( Zhou et al., 2017 ) . First, similar to linear evaluation protocol in classification, we evaluate on the fixed patch features and only fine-tune a linear layer, which gives us a more explicit comparison of the quality of representations. Second, we use the task layer in UPerNet ( Xiao et al., 2018 ) and fine-tune the entire network. From Tab. 6 , we can see that iBOT advances its supervised baseline with ViT-S/16 with a large margin of 0.9 on mIoU, surpassing Swin-T. With ViT-B/16, iBOT advances previous best methods DINO by 3.2 on mIoU with UperNet. We notice a performance drop of BEiT using linear head, indicating BEiT’s features lack local semantics. As analyzed later, the property of strong local semantics induces a 2.9 mIoU gain compared to the supervised baseline with a linear head.

##### Transfer Learning.

We study transfer learning where we pre-train on ImageNet-1K and fine-tune on several smaller datasets.We follow the training recipe and protocol used in ( Dosovitskiy et al., 2021 ) . The results are demonstrated in Tab. 7 . While the results on several datasets ( e.g . , CIFAR10, CIFAR100, Flowers, and Cars) have almost plateaued, iBOT consistently performs favorably against other SSL frameworks, achieving state-of-the-art transfer results. We observe greater performance gain over DINO in larger datasets like iNaturalist18 and iNaturalist19, indicating the results are still far from saturation. We also find that with larger models, we typically get larger performance gain compared with DINO ( e.g . , 1.7% with ViT/S-16 versus 2.0% with ViT-B/16 on iNaturalist18, and 0.3% with ViT/S-16 versus 1.0% with ViT-B/16 on iNaturalist19).

### 4.3 Properties of ViT trained with MIM

In the previous sections, we have shown the priority of iBOT on various tasks and datasets. To reveal the strengths of iBOT pre-trained Vision Transformers, we analyze its property from several aspects.

#### 4.3.1 Discovering the Pattern Layout of Image Patches

##### What Patterns Does MIM Learn?

The output from the projection head used for self-distillation depicts for patch token a probabilistic distribution. To help understand what patterns MIM induces to learn, we visualize several pattern layouts. We use 800 800 -epoch pre-trained ViT-S/16 and visualize the top- 36 36 patches with the highest confidence on ImageNet-1K validation set. We visualize a 5 × 5\times context for each 16 × 16 16\times 16 patch (colored orange). We observe the emergence of both high-level semantics and low-level details. As shown in Fig. 4 , several patches are grouped with clear semantic meaning, e.g . , headlight and dog’s ear . Such behavior stands a distinct contrast with the offline tokenizer used in BEiT ( Bao et al., 2021 ) , which encapsulates mostly low-level details as shown in Fig. 16 . Apart from patch patterns that share high-level semantics, we also observe clusters accounting for low-level textures, indicating the diversity of learned part patterns. The comparison with previous work ( Caron et al., 2021 ; Bao et al., 2021 ) and the visualization of more pattern layouts are provided in Appendix G.1 .

##### How Does MIM Help Image Recognition?

To illustrate how the property of better part semantics can help image recognition, we use part-wise linear classification to study the relationship between representations of patch tokens and [CLS] token. Specifically, we average k k patch tokens with the top- k k highest self-attention scores. The results are demonstrated in Fig. 6 . While the performance gap between DINO and iBOT is only 0.9% in the standard setting (77.9% v.s. 77.0%) with [ CLS ] [\texttt{CLS}] token, we observe that iBOT outperforms DINO when using the patch representations directly. We observe that using top- 56 56 patch tokens yields an optimal result, and iBOT is 5.9% higher than DINO. The performance gap becomes more prominent when using fewer patch tokens. When using only the patch token with the highest self-attention score, iBOT advances by 17.9%. These results reveal much semantic information in iBOT representations for patch tokens, which helps the model to be more robust to the loss of local details and further boosts its performance on image-level recognition.

#### 4.3.2 Discriminative Parts in Self-Attention Map

To analyze, we visualize the self-attention map with ViT-S/16. We choose [CLS] token as the query and visualize attention maps from different heads of the last layer with different colors, as shown in Fig. 6 . Of particular interest, we indicate that iBOT shows a solid ability to separate different objects or different parts of one object apart. For example, in the leftmost figure, we observe iBOT fairly distinct the bird from the tree branch. Also, iBOT focuses mainly on the discriminative parts of the object ( e.g . , the wheel of the car , the beak of the bird ). These properties are crucial for iBOT to excel at image recognition, especially in complicated scenarios with object occlusion or distracting instances. While these properties are not unique strengths brought by MIM and we observe similar behaviors in DINO, we show in Appendix G.2 that iBOT generally gives better visualized results.

#### 4.3.3 Robustness

The above-mentioned properties brought by MIM objective can improve the model’s robustness to uncommon examples. We quantitatively benchmark robustness in terms of 3 3 aspects: background change, occlusion, and out-of-distribution examples, with a ViT-S/16 pre-trained for 800 800 epochs and then linearly evaluated for 100 100 epochs. Results are shown in Tab. 8 . For background change, we study images under 7 7 types of change, detailed in Appendix D . iBOT is more robust against background changes except for O.BT. . For occlusion, we study the linear accuracy with salient and non-salient patch dropping following Naseer et al. (2021) with an information loss ratio of 0.5 0.5 . iBOT has a smaller performance drop under both settings. For out-of-distribution examples, we study natural adversarial examples in ImageNet-A ( Hendrycks et al., 2021 ) and image corruptions in ImageNet-C ( Hendrycks & Dietterich, 2019 ) . iBOT has higher accuracy on the ImageNet-A and a smaller mean corruptions error (mCE) on the ImageNet-C.

### 4.4 Ablation Study on Tokenizer

In this section, we ablate the importance of using a semantically meaningful tokenizer using a 300 300 -epoch pre-trained ViT-S/16 with a prediction ratio r = 0.3 r=0.3 and without multi-crop augmentation. Additional ablations are given in Appendix E . iBOT works with self-distillation on [CLS] token with cross-view images ( ℒ [CLS] \mathcal{L}_{\texttt{[CLS]}} ) to acquire visual semantics. To verify, we conduct experiments to perform MIM without ℒ [CLS] \mathcal{L}_{\texttt{[CLS]}} or with alternative models as visual tokenizer. Specifically, ∘ \circ denotes a standalone DINO and △ \triangle denotes a pre-tranined DALL-E encoder ( Ramesh et al., 2021 ) .

We find that performing MIM without ℒ [CLS] \mathcal{L}_{\texttt{[CLS]}} leads to undesirable results of 9.5% k k -NN accuracy and 29.8% linear accuracy, indicating that visual semantics can hardly be obtained with only MIM. While semantics emerges with a standalone DINO as a visual tokenizer, it is still far from reaching a decent result (44.3% versus 69.1% in k k -NN accuracy). Comparing iBOT with multi-tasking of DINO and BEiT (DINO+BEiT), we see the strengths of merging the semantics acquired by self-distillation with the visual tokenizer with an 11.5% advance in linear probing and 0.3% in fine-tuning. Moreover, we empirically observe a performance improvement using a Shared projection Head (SH) for [CLS] token and patch tokens, which shares the semantics acquired in [CLS] token to MIM.

## 5 Related Work

##### Visual Representation Learning.

Most self-supervised methods assume an augmentation invariance of images and achieve so by enforcing similarity over distorted views of one image while avoiding model collapse. Avoiding collapse can be achieved by noise-contrastive estimation with negative samples ( Wu et al., 2018 ; He et al., 2020 ; Chen et al., 2020a ) , introducing asymmetric network ( Grill et al., 2020 ; Chen & He, 2021 ) , or explicitly enforcing the distribution of image distribution over the channel to be uniform as well as one-hot ( Caron et al., 2020 ; Amrani & Bronstein, 2021 ; Caron et al., 2021 ) . In fact, the idea of simultaneously enforcing distribution uniform and one-hot is hidden from earlier studies performing representation learning via clustering ( Caron et al., 2018 ; Caron et al., 2020 ; YM. et al., 2020 ) , where the cluster assignment naturally meets these two requirements. Other methods rely on handcrafted pretext tasks and assume the image representation should instead be aware of image augmentation by solving image jigsaw puzzle ( Noroozi & Favaro, 2016 ; Wei et al., 2019 ) , predicting rotation ( Komodakis & Gidaris, 2018 ) or relative position ( Doersch et al., 2015 ) .

##### Masked Prediction in Images.

Predicting masked images parts is a popular self-supervised pretext task drawing on the idea of auto-encoding and has been previously achieved by either recovering raw pixels ( Pathak et al., 2016 ; Atito et al., 2021 ; Li et al., 2021b ) or mask contrastive learning ( Henaff, 2020 ; Zhao et al., 2021 ) . Recently, it is formulated into MIM ( Bao et al., 2021 ; Tan et al., 2021 ) with a discrete VAE ( Rolfe, 2017 ; Ramesh et al., 2021 ) as visual tokenizer. As a counterpart of MLM in NLP, MIM eases masked prediction into a classification problem supervised by labels output from the tokenizer, mitigating the problem of excessive focus on high-frequency details. Concurrently, masked image prediction has been explored in the field of multi-modality, i.e . , vision-language representation learning. These methods operate on local regions instead of global images thus reply on pre-trained detection models, i.e . , Faster-RCNN ( Ren et al., 2015 ) to propose regions of interest. ( Su et al., 2020 ; Lu et al., 2019 ; Chen et al., 2020c ) perform masked region classification tasking the category distribution output from the detection model as the ground-truth.

## 6 Conclusion