Multimodal Chain-of-Thought Reasoning in Language Models

arxiv:2302.00923

Original source · Paper JSON

Reasoning and the "physics" of language models · 2023-02-02

Cites (1) · Cited by (1)

Complete cleaned Markdown · Source Markdown including bibliography

Read in parts

1. Part 1
2. Part 2
3. Part 3

Contents

• Multimodal Chain-of-Thought Reasoning in Language Models
• Abstract
• 1 Introduction
• 2 Background
• 2.1 CoT Reasoning with LLMs
• Optimizing Demonstrations
• Optimizing Reasoning Chains
• 2.2 Eliciting CoT Reasoning by Fine-Tuning Models
• 3 Challenge of Multimodal-CoT
• 3.1 Towards the Role of CoT
• 3.2 Misleading by Hallucinated Rationales
• 3.3 Multimodality Contributes to Effective Rationales
• 4 Multimodal-CoT
• 4.1 Framework Overview
• 4.2 Model Architecture
• Encoding
• Interaction
• Decoding
• 5 Experiments
• 5.1 Dataset
• 5.2 Implementation
• Experimental Settings
• Baseline Models
• 5.3 Main Results
• 6 Analysis
• 6.1 Multimodality Boosts Convergence
• 6.2 When Multimodal-CoT Meets Large Models
• 6.3 Effectiveness Across Backbones
• 6.4 Using Different Vision Features
• 6.5 Alignment Strategies for Multimodal Interaction
• 6.6 Generalization to Other Multimodal Reasoning Benchmarks
• 6.7 Error Analysis
• 7 Conclusion
• Appendix A Extended Analysis for the Challenge of Multimodal-CoT
• A.1 Additional Examples of Misleading through Hallucinated Rationales
• A.2 Two-Stage Training Performance with Different Sizes of LMs
• A.3 Discussion of the Possible Paradigms to Achieve Multimodal-CoT
• Appendix B Experimental Details
• B.1 Details of Vision Features
• B.2 Datasets
• B.3 Implementation Details of Multimodal-CoT
• Appendix C Further Analysis
• C.1 Examples of Rationale Generation with Large Models
• C.2 Detailed Results of Multimodal-CoT on Different Backbone Models
• Appendix D Examples of Case Studies
• Instructions for reporting errors