Language Is Not All You Need: Aligning Perception with Language Models

arxiv:2302.14045

Original source · Paper JSON

LLMs: architectures, context, training · 2023-02-27

Cites (1) · Cited by (0)

Complete cleaned Markdown · Source Markdown including bibliography

Read in parts

1. Part 1
2. Part 2
3. Part 3

Contents

• Language Is Not All You Need: Aligning Perception with Language Models Thanks: Equal contribution. † \dagger Corresponding author.
• Abstract
• 1 Introduction: From LLMs to MLLMs
• From LLMs to MLLMs.
• Language models as general-purpose interfaces.
• New capabilities of MLLMs.
• 2 Kosmos-1 : A Multimodal Large Language Model
• 2.1 Input Representation
• 2.2 Multimodal Large Language Models (MLLMs)
• Magneto
• xPos
• 2.3 Training Objective
• 3 Model Training
• 3.1 Multimodal Training Data
• Text Corpora
• Image-Caption Pairs
• Interleaved Image-Text Data
• 3.2 Training Setup
• 3.3 Language-Only Instruction Tuning
• 4 Evaluation
• 4.1 Perception-Language Tasks
• 4.1.1 Evaluation Setup
• 4.1.2 Results
• Image Captioning
• Visual Question Answering
• 4.2 IQ Test: Nonverbal Reasoning
• 4.2.1 Evaluation Setup
• 4.2.2 Results
• 4.3 OCR-Free Language Understanding
• 4.3.1 Evaluation Setup
• 4.3.2 Results
• 4.4 Web Page Question Answering
• 4.4.1 Evaluation Setup
• 4.4.2 Results
• 4.5 Multimodal Chain-of-Thought Prompting
• 4.5.1 Evaluation Setup
• 4.5.2 Results
• 4.6 Zero-Shot Image Classification
• 4.6.1 Evaluation Setup
• 4.6.2 Results
• 4.7 Zero-Shot Image Classification with Descriptions
• 4.7.1 Evaluation Setup
• 4.7.2 Results
• 4.8 Language Tasks
• 4.8.1 Evaluation Setup
• 4.8.2 Results
• 4.9 Cross-modal Transfer
• 4.9.1 Transfer from Language to Multimodal: Language-Only Instruction Tuning
• 4.9.2 Transfer from Multimodal to Language: Visual Commonsense Reasoning
• Evaluation Setup
• Results
• 5 Conclusion
• Appendix A Hyperparameters
• A.1 Training
• A.2 Language-Only Instruction Tuning
• Appendix B Datasets
• B.1 Pretraning
• B.1.1 Text Corpora
• B.1.2 Image-Caption Pairs
• B.1.3 Interleaved Data
• B.2 Data Format
• Appendix C Evaluation
• C.1 Input Format Used for Perception-Language Tasks
• C.2 Language Tasks
• C.3 WebSRC Task Examples
• Instructions for reporting errors