Image as a Foreign Language: BEiT Pretraining for All Vision and Vision-Language Tasks — part 2/2

Cleaned-text code points [15582, 23578). Bibliography omitted here; source.md includes it.

The task is to measure the similarity between images and texts. There are two directions depending on the modality of the retrieved target: image-to-text retrieval, and text-to-image retrieval. Two popular retrieval benchmarks, i.e., COCO [ 31 ] , and Flickr30K [ 41 ] , are used to evaluate the model. Following previous work [ 65 , 26 ] , we use the Karpathy split [ 24 ] for the two benchmarks. BEiT-3 is finetuned as a dual encoder for efficient image-text retrieval. Dual-encoder models separately encode images and texts to obtain their representations. Then we calculate the cosine similarity scores of these representations. Dual-encoder models are more efficient than fusion-encoder models. Because they do not have to jointly encode all possible image-text pairs.

We directly finetune BEiT-3 on COCO and Flickr30K, although the model is not pretrained with image-text contrastive loss. Surprisingly, BEiT-3 outperforms previous state-of-the-art models only using a small amount of contrastive training. The results demonstrate that BEiT-3 effectively learns alignments between images and texts via masked data modeling. In order to improve the performance, we perform intermediate finetuning with an image-text contrastive objective on the pretraining image-text pairs. We finetune the model with much fewer steps than pretraining. Then we use the model to evaluate zero-shot and finetuned image-text retrieval. The finetuned results are present in Table 5 , dual-encoder BEiT-3 outperforms prior models by a large margin, achieving 3.0 3.0 / 4.0 4.0 absolute improvement on COCO top- 1 1 image-to-text/text-to-image retrieval, and 0.8 0.8 / 2.4 2.4 absolute improvement on Flickr30K top- 1 1 image-to-text/text-to-image retrieval. BEiT-3 also significantly outperforms fusion-encoder-based models, which require more computation cost for inference. As present in Table 6 , BEiT-3 also achieves better performance than previous models on Flickr30K zero-shot retrieval.

### 3.2 Vision Downstream Tasks

In addition to vision-language downstream tasks, BEiT-3 can be transferred to a wide range of vision downstream tasks, including object detection, instance segmentation, semantic segmentation, and image classification. The number of effective parameters is comparable to ViT-giant [ 63 ] , i.e., about 1B, when BEiT-3 is used as a vision encoder.

#### Object Detection and Instance Segmentation

We conduct finetuning experiments on the COCO 2017 benchmark [ 31 ] , which consists of 118 118 k training, 5 5 k validation, and 20 20 k test-dev images. We use BEiT-3 as the backbone and follow ViTDet [ 32 ] , including a simple feature pyramid and window attention, for the object detection and instance segmentation tasks. Following common practices [ 29 , 66 ] , we first conduct intermediate finetuning on the Objects365 [ 48 ] dataset. Then we finetune the model on the COCO dataset. Soft-NMS [ 6 ] is used during inference. Table 7 compares BEiT-3 with previous state-of-the-art models on COCO object detection and instance segmentation. BEiT-3 achieves the best results on the COCO test-dev set with a smaller image size used for finetuning, reaching up to 63.7 63.7 box AP and 54.8 54.8 mask AP.

#### Semantic Segmentation

Semantic segmentation aims to predict the label for each pixel of the given image. We evaluate BEiT-3 on the challenging ADE20K dataset [ 68 ] , which includes 150 150 semantic categories. ADE20K contains 20 20 k images for training and 2 2 k images for validation. We directly follow the task transfer settings of ViT-Adapter [ 8 ] . We use a dense prediction task adapter and employ Mask2Former [ 10 ] as the segmentation framework. As shown in Table 8 , BEiT-3 creates a new state-of-the-art result with 62.8 62.8 mIoU, outperforming FD-SwinV2 [ 54 ] giant model with 3B parameters by 1.4 1.4 points. It shows that BEiT-3 achieves superior performance on the dense prediction task.

#### Image Classification

We evaluate the model on ImageNet-1K [ 42 ] , which contains 1.28 1.28 M training images and 50 50 k validation images in 1 1 k classes. Rather than appending a task layer to the vision encoder [ 13 , 3 ] , we formulate the task as an image-to-text retrieval task. We use the category names as texts to construct image-text pairs. BEiT-3 is trained as a dual encoder to find the most relevant label for an image. During inference, we first compute the feature embeddings of possible class names and the feature embedding of the image. Their cosine similarity scores are then calculated to predict the most probable label for each image. Table 9 reports the results on ImageNet-1K. We first perform intermediate finetuning on ImageNet-21K, then we train the model on ImageNet-1K. For a fair comparison, we compare with the previous models only using public image-tag data. BEiT-3 outperforms prior models, creating a new state-of-the-art result when only using public image-tag data.

## 4 Conclusion

In this paper, we present BEiT-3 , a general-purpose multimodal foundation model, which achieves state-of-the-art performance across a wide range of vision and vision-language benchmarks. The key idea of BEiT-3 is that image can be modeled as a foreign language, so that we can conduct masked “language” modeling over images, texts, and image-text pairs in a unified way. We also demonstrate that Multiway Transformers can effectively model different vision and vision-language tasks, making it an intriguing option for general-purpose modeling. BEiT-3 is simple and effective, and is a promising direction for scaling up multimodal foundation models. For future work, we are working on pretraining multilingual BEiT-3 and including more modalities (e.g., audio) in BEiT-3 to facilitate the cross-lingual and cross-modality transfer, and advance the big convergence of large-scale pretraining across tasks, languages, and modalities. We are also interested in enabling in-context learning capability for multimodal foundation models by combining the strength of BEiT-3 and MetaLM [ 20 ] .

## Appendix A Effects of Intermediate Finetuning for Retrieval

As shown in Table 10 , we directly finetune BEiT-3 on COCO and Flickr30K. BEiT-3 still outperforms previous state-of-the-art models, even without using image-text contrastive objective during pretraining. The results demonstrate the effectiveness of masked data modeling for learning cross-modal representations. Next, we perform intermediate finetuning on the pretraining image-text pairs for 5 5 epochs with a 16 16 k batch size. The peak learning is 3e-5, with linear warmup over the first epoch. The image input size is 224 × 224 224\times 224 . The weight decay is set to 0.05 0.05 . We disable dropout as in pretraining and use drop path with a rate of 0.3 0.3 . The layer-wise learning rate decay is 0.95 0.95 . We use the AdamW [ 28 ] optimizer with β 1 = 0.9 \beta_{1}=0.9 , β 2 = 0.999 \beta_{2}=0.999 .

## Appendix B Hyperparameters Used for Pretraining

## Appendix C Hyperparameters Used for Finetuning

## Instructions for reporting errors

We are continuing to improve HTML versions of papers, and your feedback helps enhance accessibility and mobile support. To report errors in the HTML that will help us improve conversion and rendering, choose any of the methods listed below:

Click the "Report Issue" ( ) button, located in the page header.

Tip: You can select the relevant text first, to include it in your report.

Our team has already identified the following issues . We appreciate your time reviewing and reporting rendering errors we may not have found yet. Your efforts will help us improve the HTML versions for all readers, because disability should not be a barrier to accessing research. Thank you for your continued support in championing open access for all.

Have a free development cycle? Help support accessibility at arXiv! Our collaborators at LaTeXML maintain a list of packages that need conversion , and welcome developer contributions .