iBOT: Image BERT Pre-Training with Online Tokenizer — part 4/4

Cleaned-text code points [47125, 60722). Bibliography omitted here; source.md includes it.

We study the model’s sensitivity to the spatial structure by shuffling on input image patches. Specifically, we shuffle the image patches with different grid sizes following ( Naseer et al., 2021 ) . We showcase the results of iBOT end-to-end fine-tuned or with a linear head over the pre-trained backbone. We include the results of supervised results with both ViT-S/16 and ResNet-50 for comparison. Note that a shuffle grid size of 1 1 means no shuffle, and a shuffle grid size of 196 196 means all patch tokens are shuffled. Fig. 10 suggests that iBOT retain accuracy better than its supervised baseline and ResNet-50. It also indicates that iBOT relies less on positional embedding to preserve the global image context for right classification decisions.

## Appendix E Additional Ablations

In this section, we study the impact of other parameters that we have conducted experiments on. Without extra illustrations, we use 300 300 -epoch pre-trained ViT-S/16, a prediction ratio r = 0.3 r=0.3 and without multi-crop augmentation for the ablative study.

##### Architecture of Projection Head.

As mentioned earlier, a shared head can transfer the semantics acquired in [CLS] token to patch tokens, slightly improving the performance. We notice that the head for patch tokens in the student network only see the masked tokens throughout the training, the distribution of which mismatches tokens with natural textures. Therefore, we conduct an experiment using a non-shared head for the student network but a shared head for the teacher network denoted as semi-shared head . Their differences are demonstrated in Fig. 18 , where S and T denotes student and teacher network respectively. The heads with the same index and color denotes they have shared parameters.

† denotes only the first 2 2 layers out of the 3 3 -layer MLP share the parameters. However, we do not observe that semi-shared head is better than shared head. By default, we share the entire projection head for [CLS] token and patch tokens.

##### Comparing MIM with Dense Self-Distillation.

To identify the superiority of MIM to model internal structure using over its alternatives, we conduct experiments performing self-distillation on original patch tokens along with the [CLS] token. We consider two matching strategies to construct patch token pairs for self-distillation.

Specifically, pos. denotes matching according to the absolute position of two views. Similar to Xie et al. (2021b) . j j is defined as arg ⁡ min j ⁡ d ​ i ​ s ​ t ​ ( p i , p j ′ ) \mathop{\arg\min}_{j}dist(p_{i},p^{\prime}_{j}) , where p p is the position in the original image space and d ​ i ​ s ​ t ​ ( u , v ) dist(u,v) is euclidean distance. The losses are only computed for the overlapped regions of two views. We do not observe substantial gain brought by matching via patches’ absolute position. feat. denotes matching according to the similarity of the backbone similarity of two views. Similar to Wang et al. (2021b) , we match for each patch token f i f_{i} the most similar patch token from another view f j ′ f^{\prime}_{j} , where j = arg ⁡ max j ⁡ s ​ i ​ m ​ ( f i , f j ′ ) j=\mathop{\arg\max}_{j}sim(f_{i},f^{\prime}_{j}) . s ​ i ​ m ​ ( u , v ) sim(u,v) is cosine distance. Such practice brings a 0.6 % 0.6\% performance gain in terms of linear probing accuracy, which is also observed by a concurrent work, EsViT ( Li et al., 2021a ) . Comparatively, iBOT prompts an 1.2 % 1.2\% gain on linear probing, verifying the necessity and advancement of MIM.

##### Hard Label versus Soft Label

We study the importance of using a continuous token distribution (softmax † ) instead of a discretized id (hardmax) when performing MIM. Results in Tab. 18 indicate continuous tokenization plays a crucial part. We empirically find the improvement brought by centering, whose roles are less important compared to centering in self-distillation on [CLS] token. Only sharpening can produce a k k -NN accuracy of 69.4 and a linear probing accuracy of 73.9.

##### Centering and Sharpening.

Different from the [CLS] token, patch tokens do not have certain semantic cluster and vary more widely from each others. We study the impact of several critical parameters that decide the distillation process and customize them for distillation over the patch tokens.

Specifically, the smoothing momentum for online centering m ′ m^{\prime} and sharpening temperature τ t ′ \tau^{\prime}_{t} are studied. Note we keep the parameters for [CLS] token the same as DINO and only study for parameters for the patch tokens.

##### Loss Ratio.

We study the impact of different ratio between ℒ [CLS] \mathcal{L}_{\texttt{[CLS]}} and ℒ MIM \mathcal{L}_{\mathrm{MIM}} . We keep the base of ℒ [CLS] \mathcal{L}_{\texttt{[CLS]}} to 1 1 and scale ℒ MIM \mathcal{L}_{\mathrm{MIM}} with different ratios.

We observe that directly adding two losses up without scaling yields the best performance in terms of linear probing accuracy.

##### Output Dimension.

We follow the structure of projection head in DINO with l2-normalized bottleneck and without batch normalization. We study the impact of output dimension K K of the last layer.

While our method excludes large output dimensionality since each patch token has an output distribution, we do not observe substantial performance gain brought by larger output dimensions. Therefore, we choose K = 8192 K=8192 by default.

##### Prediction Ratios.

Masked modeling is based on a formulation of partial prediction, the objective of which is to maximize the log-likelihood of the target tokens conditioned on the non-target tokens. We experiment with different prediction ratios for masked image modeling. The results are shown in Fig. 12 . We observe that the performance is not sensitive to variant prediction ratios between 0.05 0.05 and 0.4 0.4 . Adding a variance upon the fixed value can also consistently bring a performance gain, which can be explained as stronger data augmentation. The teacher output of non-masked images is now pulled together with the student output of masked images with different ratios. By default, we use 0.3 ​ ( ± 0.2 ) 0.3\ (\pm 0.2) as the prediction ratio. For models with multi-crop augmentation, following the above discussions, we randomly choose a prediction of 0 0 or 0.3 ​ ( ± 0.2 ) 0.3\ (\pm 0.2) for each image.

##### Training Epochs.

We provide the linear probing top-1 accuracy with ViT-S/16 pre-trained for different epochs. For comparison, we also include the accuracy curve of other methods with comparable numbers of parameters, i.e . , ResNet-50. From Fig. 12 , we observe that longer training for 800 800 epochs can improve the model’s performance. It’s north worthy that iBOT can achieve a Top-1 accuracy of SwAV ( Caron et al., 2020 ) pre-trained with 800 800 epochs in less than 100 100 epochs. iBOT pre-trained with 800 800 epochs brings a 0.9% improvement over previous state-of-the-art method.

##### Time and Memory Requirements.

BEiT is trained with a non-contrastive objective and without multi-crop augmentation, thus it consumes only a memory of 5.6G and takes 90.1h for 800 800 epochs. Comparing iBOT and DINO with multi-crop augmentation, iBOT with MIM induces 25 % 25\% more memory requirements and 7.4 % 7.4\% more actual training time. Considering pre-training efficiency (accuracy versus time), 800-epochs pre-trained DINO requiring for 180.0h, while 300-epochs iBOT only requires 73.3h with 0.4% higher linear probing accuracy (77.0 versus 77.4).

## Appendix F Alternative Tokenizers

To investigate how different approaches to tokenize the patches affect MIM, we study several alternatives. In BEiT ( Bao et al., 2021 ) , masked patches are tokenized by a DALL-E encoder. MPP ( Dosovitskiy et al., 2021 ) tokenizes the masked patches using their 3-bit mean color. For Patch Clustering, we first perform K K -Means algorithm to the flattened color vector of each 16 × 16 16\times 16 patch ( d = 768 d=768 ). 10 % 10\% data of ImageNet-1K training set is sampled and clustered. We set K K to 4096 4096 . During pre-training, each patch is tokenized by the index of its closest centroids. Lastly, we use 300 300 -epoch pre-trained DINO as a standalone tokenizer. Each patch can be tokenized by the argmax of its output from the pre-trained DINO. We use average pooling to aggregate the patch representations. From Tab. 20 , we see that all methods achieve decent fine-tuning results compared to the supervised baseline, while only methods tokenized by semantically meaningful tokenizer have proper results on k k -NN and linear classification. MPP ( Dosovitskiy et al., 2021 ) and patch clustering rely purely on offline statistics without the extra stage of online training. We find patch clustering has slightly better performance in all three protocols compared to MPP, suggesting the benefits brought by visual semantics. While BEiT has poor k k -NN and linear probing accuracy, a good fine-tuning result also suggests relatively low requirements for fine-tuning protocol on high-level semantics.

## Appendix G Visualization

In this section, we first give more visualized pattern layouts and self-attention maps. Beyond that, we consider an additional task of mining sparse correspondences between two images and illustrating the superiority of ViTs by showcasing several visualized results.

### G.1 Pattern Layout

##### Pattern Layout for Patch Tokens.

To illustrate versatile, interesting behaviors iBOT has learned, we organize the visualization of pattern layout in two figures. In Fig. 13 , we mainly showcase additional pattern layouts that share high-level semantics. In Fig. 14 , we mainly showcase additional pattern layouts that share low-level details like color, texture, shape, etc . Top 100 100 patches with the highest confidence over the validation set are visualized with a 5 × 5 5\times 5 context around each 16 × 16 16\times 16 patch token (colored orange).

##### Composing Images with Representative Patterns.

In Fig. 15 , we visualize 4 4 patches with the highest self-attention score (with non-overlapped assigned index) and also show the pattern layout of that assigned index. The visualized results indicate iBOT can only be represented by several representative patches, which helps the model’s robustness and performance in recognition. This is also validated by our part-wise linear probing experiments.

##### Comparison with Other Methods.

We visualize pattern layout for patch tokens using other self-supervised methods ( Bao et al., 2021 ; Caron et al., 2021 ) in Fig. 16 . For BEiT, the DALL-E encoder generates a discrete number for each patch token. For DINO, we directly use the projection head for [CLS] token and generate a 65536 65536 -d probability distribution for each patch token. The index with the highest probability is assigned for the token.

##### Pattern Layout for [CLS] Token.

We here also provide additional visualization of semantic patterns emerge in [CLS] token, which is obtained via self-distillation on cross-view images. We also observe similar behavior in DINO since it’s not a unique property brought by MIM. In fact, semantics are now believed to emerge as long as a similarity between two distorted views of one image is enforced ( Grill et al., 2020 ; He et al., 2020 ; Caron et al., 2020 ; Caron et al., 2018 ) .

### G.2 Self-Attention Visualizations

Similar to the setting of Sec. 4.3.2 , we here provided more self-attention map visualization from multiple heads of the last layer in Fig. 18 .

### G.3 Sparse Correspondence.

We consider a sparse correspondence task where the overlapped patches from two augmented views of one image, or patches from two images labeled as one class, are required to be matched. The correlation is sparse since at most 14 × 14 14\times 14 matched pairs can be extracted with a ViT-S/16 model. We visualize 12 12 correspondences with the highest self-attention score extracted from iBOT with ViT-S/16 pre-trained for 800 800 epochs. The score is averaged between multiple heads of the last layer. Several sampled sets of image pairs are shown in Fig. 19 . We observe empirically that iBOT perform well for two views drawn from one image, nearly matched the majority of correspondence correctly. In the second column, iBOT can match different parts of two instances from the same class ( e.g . , tiles and windows of two cars) despite their huge differences in texture or color. We observe the DINO also has comparable visualized effects, illustrating the representation pre-trained with self-distillation also suits well for retrieval in a patch-level scale.

## Instructions for reporting errors

We are continuing to improve HTML versions of papers, and your feedback helps enhance accessibility and mobile support. To report errors in the HTML that will help us improve conversion and rendering, choose any of the methods listed below:

Click the "Report Issue" ( ) button, located in the page header.

Tip: You can select the relevant text first, to include it in your report.

Our team has already identified the following issues . We appreciate your time reviewing and reporting rendering errors we may not have found yet. Your efforts will help us improve the HTML versions for all readers, because disability should not be a barrier to accessing research. Thank you for your continued support in championing open access for all.

Have a free development cycle? Help support accessibility at arXiv! Our collaborators at LaTeXML maintain a list of packages that need conversion , and welcome developer contributions .