iBOT: Image BERT Pre-Training with Online Tokenizer — part 3/4

Cleaned-text code points [31218, 47125). Bibliography omitted here; source.md includes it.

In this work, we study BERT-like pre-training for Vision Transformers and underline the significance of a semantically meaningful visual tokenizer. We present a self-supervised framework iBOT that performs masked image modeling via self-distillation with an online tokenizer, achieving state-of-the-art results on downstream tasks related to classification, object detection, instance segmentation, and semantic segmentation. Of particular interest, we identify an emerging part-level semantics for models trained with MIM that helps for not only recognition accuracy but also robustness against common image corruptions. In the future, we plan to scale up iBOT to a larger dataset ( e.g . , ImageNet-22K) or larger model size ( e.g . , ViT-L/16 and ViT-H/16) and investigate whether MIM can help Vision Transformers more scalable to unlabelled data in the wild.

Acknowledgement Tao Kong is the corresponding author. We would like to acknowledge Feng Wang, Rufeng Zhang, and Zongwei Zhou for helpful discussions. We thank Mathilde Caron, Julien Mairal, and Hugo Touvronfor for sharing details of DINO. We thank Li Dong and Hangbo Bao for sharing details of BEiT.

## Appendix A Pseudocode

## Appendix B Multi-Crop

The advanced performance of several recent state-of-the-art methods ( Caron et al., 2021 ; Caron et al., 2020 ) relies on multi-crop augmentation, as well as iBOT. In our early experiments, we find the direct usage of multi-crop augmentation leads to instability issues that degrade accuracy. We reveal that these results can be attributed to the distribution mismatch between masked images and non-masked images and can be resolved by minimal changes in iBOT framework.

##### Stability of MIM Pre-trained with Multi-Crop.

We first showcase several practices where training instability occurs, shown in Fig. 7 . To reveal the instability, we monitor the NMI curves during training for each epoch as shown in Fig. 8 . The most intuitive ideas are to compute as (b) or (c). In (b), MIM is only performed on global crops. This pipeline is unstable during training, and we observe a dip in the NMI training curve. We hypothesize that it can be caused by the distribution mismatch of masked global crops and non-masked local crops. To alleviate this, a straightforward solution is to also perform MIM on local crops with an extra computation cost as (c). However, we do not observe this circumvents training instability. We hypothesize that the regions corresponding to patch tokens of the local crops are small in size, in which there exist few meaningful contents to predict. This hypothesis can be supported by the experiments that when we set the local crop scale in (c) from ( 0.05 , 0.4 ) (0.05,0.4) to ( 0.2 , 0.4 ) (0.2,0.4) , denoted as (e), the performance drop is mitigated.

##### Stabilizing the Training with Non-Masked Global Crops.

Another solution to alleviate the distribution mismatch between masked global crops and non-masked local crops is to train with non-masked global crops, as shown in (d). In other words, we perform random MIM when training ViT with multi-crop augmentation. This computation pipeline is stable and achieves a substantial performance gain. In practice, to include non-masked global crops in training, we use (b) and randomly choose a prediction ratio between [ 0 0 , r ⁡ ( r > 0 ) r\ (r>0) ] for each image. When the ratio 0 0 is chosen, the whole framework excludes MIM and can be seen as DINO. When the ratio r ⁡ ( r > 0 ) r\ (r>0) is chosen, MIM is performed for both of the two global crops. We observe the latter practice performs sightly better since it is more flexible in task composition and data in a batch is mutually independent.

##### Range of Scales in Multi-Crop.

We further study the performance with different local and global scale. Following DINO ( Caron et al., 2021 ) , we conduct the experiments by tweaking s s , where s s is the scale deviding the local and global crops. The local crops are sampled from (0.05, s s ) and the global crops are sampled from ( s s , 1).

We empirically find that s = 0.32 s=0.32 yields optimal performance for both small-size and base-size models. Therefore, we use an s s of 0.32 0.32 by default.

##### State-of-the-Art Comparison w/o and w/ Multi-Crop.

Including iBOT, several recent state-of-the-art works ( Caron et al., 2021 ; Caron et al., 2020 ) rely heavily on multi-crop augmentation during pre-training. Except for several specific self-supervised methods ( Grill et al., 2020 ) , multi-crop works well on most of the self-supervised methods and consistently yields performance gain ( Caron et al., 2021 ) . While a more fair comparison with our methods without multi-crop augmentation can be conducted, we believe it is a unique strength of iBOT to work well with multi-crop. In Tab. 10 , we categorize the state-of-the-art comparison into two parts where one for methods without multi-crop and the other with multi-crop. For the former, we mainly compare our method without multi-crop with MoCov3 ( Chen et al., 2021 ) and DINO without multi-crop. We observe that our method achieves state-of-the-art performance with ViT-S/16 even without multi-crop and comparable performance with ViT-B/16 compared with MoCov3. For the latter, we mainly compare our method with SwAV ( Caron et al., 2020 ) and DINO with multi-crop augmentation. We observe that iBOT achieves higher performance with 79.4% of linear probing accuracy when using ViT-S/16.

##### Effective Training Epochs.

Due to extra computation costs brought by multi-crop augmentation, different methods with the same pre-training epochs actually see different total numbers of images. To mitigate, we propose to measure the effective training epochs, defined as actual pre-training epochs multiplied with a scaling factor accounting for extra trained images of different resolutions induced by multi-crop augmentation. DINO and iBOT are by default trained with 2 2 global crops of size 224 × 224 224\times 224 and 10 10 local crops of size 96 × 96 96\times 96 . Thus r = 2 + ( 96 224 ) 2 × 10 = 3.84 ≈ 4 r=2+(\frac{96}{224})^{2}\times 10=3.84\approx 4 for DINO and iBOT. r ≈ 3 r\approx 3 for SwAV or DINO with RN50 as the backbone and pre-trained with 2 2 global crops and 6 6 local crops. r = 2 r=2 for contrastive methods without multi-crop augmentation ( e.g . , MoCo, SimCLR, BYOL, etc . ) and r = 1 r=1 for non-contrastive methods ( e.g . , BEiT, Jigsaw, etc . ).

## Appendix C Additional Implementations

##### Fine-Tuning Recipes of Classification on ImageNet-1K.

By default, we follow the fine-tuning protocol in BEiT ( Bao et al., 2021 ) to use a layer-wise learning rate decay, weight decay and AdamW optimizer and train small-, base-size models with 200 200 , 100 100 , and 50 50 epochs respectively. We sweep over four learning rates { 8 ​ e − 4 , 9 ​ e − 4 , 1 ​ e − 3 , 2 ​ e − 3 } \{8e^{-4},9e^{-4},1e^{-3},2e^{-3}\} . Comparatively, traditional fine-tuning recipe is is to fine-tune the network for 300 300 epochs with a learning rate 5 ​ e − 4 5e^{-4} , no weight decay, and SGD optimizer ( Touvron et al., 2021 ) (Row 1 versus 8 ). For a fair comparison, we compare the impact of different fine-tuning recipes with different methods, shown in Tab. 12 . We empirically find that fine-tuning protocol used in BEiT consistently yields better fine-tuning results and greatly reduces the training epochs. By default, we use a layerwise decay of 0.75 0.75 with a training epoch of 200 200 for ViT-S/16, a layerwise decay of 0.65 0.65 with a training epoch of 100 100 for ViT-B/16, and a layerwise decay of 0.75 0.75 with a training epoch of 50 50 for ViT-L/16. We report the higher results between using or not using DS since we find it brings different impacts to different methods.

##### Evaluation Protocols of Semi-Supervised Learning on ImageNet-1K.

We study the impact of different evaluation protocols for semi-supervised learning. Under conventional semi-supervised evaluation protocol, pre-trained models are end-to-end fine-tuned with a linear classification head. SimCLRv2 Chen et al. (2020b) found that keeping the first layer of the projection head can improve accuracy, especially under the low-shot setting. We fine-tune the pre-trained model from the first layer of the projection head and verify this conclusion holds true for Vision Transformers. We empirically find that Vision Transformer performs better with a frozen backbone with 1 % 1\% of training data (62.5% in row 4 versus 61.9 % in row 7 ). In DINO, a logistic regressor built upon the frozen features is found to perform better compared with the multi-class linear classifier upon the frozen features, especially with 1 % 1\% data (65.9% in row 6 versus 62.5% in row 4 ). When using 10 % 10\% data, we empirically find that end-to-end fine-tuning from the first layer of the projection layer yields the best performance (75.1% in row 10 versus 73.4% in row 6 ).

##### Fine-Tuning Recipes of Object Detection and Instance Segmentation on COCO.

For both small- and base-size models, we utilize multi-scale training (resizing image with shorter size between 480 480 and 800 800 while the longer side no larger than 1333 1333 ), a learning rate 1 ​ e − 4 1e^{-4} , a weight decay of 0.05 0.05 , and fine-tune the entire network for 1 × 1\times schedule (12 epochs with the learning rate decayed by 10 × 10\times at epochs 9 9 and 11 11 ). We sweep a layer decay rate of { 0.65 0.65 , 0.75 0.75 , 0.8 0.8 , 0.9 0.9 }. Note that a layer decay rate of 1.0 1.0 denotes no layer is decayed. To produce hierarchical feature maps, we use the features output from layer 4 4 , 6 6 , 8 8 , and 12 12 , with 2 2 deconvolutions, 1 1 deconvolution, identity mapping, and max-pooling appended after, respectively. We do not use multi-scale testing.

##### Fine-Tuning Recipes of Semantic Segmentation on ADE20K.

For semantic segmentation, we follow the configurations in BEiT ( Bao et al., 2021 ) , fine-tuning 160 160 k iterations with 512 × 512 512\times 512 images and a layer decay rate of 0.65 0.65 . We do not use multi-scale training and testing. We sweep the learning rate { 3 ​ e − 5 , 8 ​ e − 5 , 1 ​ e − 4 , 3 ​ e − 4 , 8 ​ e − 4 } \{3e^{-5},8e^{-5},1e^{-4},3e^{-4},8e^{-4}\} . Similar to object detection and instance segmentation, to produce hierarchical feature maps, we add additional deconvolution layers after ViT.

When using linear (Lin.) as the task layer, we find that appending the last LayerNorm ( [LN] ) for [CLS] token to each patch tokens before the decoder consistently yields better performance, while we do not spot the substantial gain when with UperNet as the task layer. By default, we report the segmentation result with [LN] for both linear head for UperNet head.

##### Part-Wise Linear Probing.

We use the average of the last-layer self-attention map with [CLS] as the query from multiple heads to rank all the patch tokens. We remove the extra LayerNorm (LN) after the final block following MoCov3 ( Chen et al., 2021 ) .

## Appendix D Additional Results

In this section, we provide detailed results for dense downstream tasks, i.e . , object detection, instance segmentation, and semantic segmentation. We give the complete figures for occlusion robustness analysis. We also provide extra experiments of nearest neighbor retrieval, robustness analysis against occlusion and shuffle.

##### Object Detection, Instance Segmentation, and Semantic Segmentation.

We here provide more detailed results on object detection, instance segmentation, and semantic segmentation with small- and base-size models, shown in Tab. 13 and Tab. 14 respectively. Specifically, we include AP 50 b {}^{\mathrm{b}}_{\mathrm{50}} and AP 75 b {}^{\mathrm{b}}_{\mathrm{75}} for object detection, AP 50 m {}^{\mathrm{m}}_{\mathrm{50}} and AP 75 m {}^{\mathrm{m}}_{\mathrm{75}} for instance segmentation, mAcc for semantic segmentation. For object detection (Det.) and instance segmentation (Inst. Seg.), we consider Cascade Mask R-CNN as the task layer. For semantic segmentation (Seg.), we consider two evaluation settings where a linear head (Lin.) and UPerNet are taken as the task layer.

##### k k -NN and Linear Probing with ImageNet-22K.

We further report k k -NN and linear probing accuracy on ImageNet-1K with models pre-trained on ImageNet-22K dataset. We empirically observe that ImageNet-1K pre-training incurs better ImageNet-1K k k -NN and linear probing performance, which is opposite to the fine-tuning performance observed in Table 3 and Table 3 . We hypothesize that the data distribution plays a more crucial rule under evaluation protocols based on frozen features, such that models pre-trained with smaller ImageNet-1K dataset consistently achieve better results.

##### Nearest Neighbor Retrieval.

Nearest neighbor retrieval is considered using the frozen pre-trained features following the evaluation protocol as in DINO ( Caron et al., 2021 ) . DINO has demonstrated the strong potential of pre-trained ViT features to be directly used for retrieval. To validate, DINO designed several downstream tasks, including image retrieval and video object segmentation, where video object segmentation can be seen as a dense retrieval task by finding the nearest neighbor between consecutive frames to propagate masks. We compare iBOT with DINO on these benchmarks with the same evaluation settings. As demonstrated in Tab. 16 , iBOT has comparable results with DINO. While iBOT has higher k k -NN results on Imagenet-1K, the performance is not better for iBOT in image retrieval. We empirically find that the results on image retrieval are sensitive to image resolution, multi-scale features, etc . , and the performance varies using pre-trained models with minimal differences on hyper-parameter setup. For this reason, we do not further push iBOT for better results.

##### Robustness against Background Change.

Deep models rely on both foreground objects and backgrounds. Robust models should be tolerant to background changes and able to locate discriminative foreground parts. We evaluate this property on ImageNet-9 (IN-9) dataset ( Xiao et al., 2020 ) . IN-9 includes 9 9 coarse-grained classes and 7 7 variants by mixing up the foreground and background from different images. Only-FG (O.F.) , Mixed-Same (M.S.) , Mixed-Rand (M.R.) , and Mixed-Next (M.N.) are 4 4 variant datasets where the original foreground is present but the background is modified, whereas No-FG (N.F.) , Only-BG-B (O.BB.) , and Only-BG-T (O.BT.) are 3 3 variants where the foreground is masked. As shown in Tab. 8 , we observe a performance gain except for O.BT. , indicating iBOT’s robustness against background changes. We note in O.BT. neither foreground nor foreground mask is visible, contradicting the pre-training objective of MIM.

##### Robustness against Occlusion.

Masked prediction has a natural strength in cases where parts of the image are masked out since the models are trained to predict their original contents. We here provide the detailed results of occlusion with different information loss ratios in Fig. 9 under three dropping settings: random, salient, and non-salient. We showcase the results of iBOT end-to-end fine-tuned or with a linear head over the pre-trained backbone. We include the results of supervised results with both ViT-S/16 and ResNet-50 for comparison. ViT shows higher robustness compared to its CNN counterpart, i.e . , ResNet-50, given that Transformers’ dynamic receptive field makes it less dependent on images’ spatial structure. We empirically find iBOT has stronger robustness against occlusion compared to its supervised baseline, consolidating that MIM help to model the interaction between the sequence of image patches using self-attention such that discarding proportion of elements does not degrade the performance significantly.

##### Robustness against Shuffle.