Image as a Foreign Language: BEiT Pretraining for All Vision and Vision-Language Tasks

arxiv:2208.10442

Original source · Paper JSON

Self-supervised learning and vision · 2022-08-22

Cites (0) · Cited by (2)

Complete cleaned Markdown · Source Markdown including bibliography

Read in parts

1. Part 1
2. Part 2

Contents

• Image as a Foreign Language: BEiT Pretraining for All Vision and Vision-Language Tasks
• Abstract
• 1 Introduction: The Big Convergence
• 2 BEiT-3 : A General-Purpose Multimodal Foundation Model
• 2.1 Backbone Network: Multiway Transformers
• 2.2 Pretraining Task: Masked Data Modeling
• 2.3 Scaling Up: BEiT-3 Pretraining
• Backbone Network
• Pretraining Data
• Pretraining Settings
• 3 Experiments on Vision and Vision-Language Tasks
• 3.1 Vision-Language Downstream Tasks
• Visual Question Answering (VQA)
• Visual Reasoning
• Image Captioning
• Image-Text Retrieval
• 3.2 Vision Downstream Tasks
• Object Detection and Instance Segmentation
• Semantic Segmentation
• Image Classification
• 4 Conclusion
• Appendix A Effects of Intermediate Finetuning for Retrieval
• Appendix B Hyperparameters Used for Pretraining
• Appendix C Hyperparameters Used for Finetuning
• Instructions for reporting errors