Multimodal Chain-of-Thought Reasoning in Language Models — part 3/3

Cleaned-text code points [31801, 47547). Bibliography omitted here; source.md includes it.

To test the generality of the benefits of our approach to other backbone models, we alter the underlying LMs to other variants in different types. As shown in Table 6.3 , our approach is generally effective for the widely used backbone models.

### 6.4 Using Different Vision Features

Different vision features may affect the model performance. We compare three widely-used types of vision features, ViT ( Dosovitskiy et al., 2021b ) , CLIP ( Radford et al., 2021 ) , DETR ( Carion et al., 2020 ) , and ResNet ( He et al., 2016 ) . ViT, CLIP, and DETR are patch-like features. For the ResNet features, we repeat the pooled features of ResNet-50 to the same length with the text sequence to imitate the patch-like features, where each patch is the same as the pooled image features. More details of the vision features are presented in Appendix B.1 .

Table 6.3 shows the comparative results of vision features. We observe that ViT achieves relatively better performance. Therefore, we use ViT by default in Multimodal-CoT.

### 6.5 Alignment Strategies for Multimodal Interaction

We are interested in whether using different alignment strategies for multimodal interaction may contribute to different behaviors of multimodal-CoT. To this end, we tried another alignment strategy, i.e., image-grounded text encoder, in BLIP Li et al. (2022b) . This alignment approach injects visual information by inserting one additional cross-attention layer between the self-attention layer and the feed-forward network for each transformer block of the text encoder. Our current strategy in the paper is similar to the unimodal encoder as in BLIP, which is used for comparison. In Table 10 , we see that using other alignment strategies also contributes to better performance than direct answering.

### 6.6 Generalization to Other Multimodal Reasoning Benchmarks

We are interested in evaluating the generalization capability of Multimodal-CoT to datasets outside its training domain. For this purpose, we utilize the widely-recognized multimodal reasoning benchmark, MMMU ( Yue et al., 2024 ) , and conduct an evaluation of Multimodal-CoT on MMMU without further training.

As shown in Table 11 , it is evident that Multimodal-CoT demonstrates effective generalization to MMMU, achieving better performance than various larger models around 8B.

### 6.7 Error Analysis

To gain deeper insights into the behavior of Multimodal-CoT and facilitate future research, we manually analyzed randomly selected examples generated by our approach. The categorization results are illustrated in Figure 6 . We examined 50 samples that yielded incorrect answers and categorized them accordingly. The examples from each category can be found in Appendix D .

The most prevalent error type is commonsense mistakes, accounting for 80% of the errors. These mistakes occur when the model is faced with questions that require commonsense knowledge, such as interpreting maps, counting objects in images, or utilizing the alphabet. The second error type is logical mistakes, constituting 14% of the errors, which involve contradictions in the reasoning process. Additionally, we have observed cases where incorrect answers are provided despite the CoT being either empty or correct, amounting to 6% of the errors. The CoT in these cases may not necessarily influence the final answer.

The analysis reveals potential avenues for future research. Enhancements can be made to Multimodal-CoT by: (i) integrating more informative visual features and strengthening the interaction between language and vision to enable comprehension of maps and numerical counting; (ii) incorporating commonsense knowledge; and (iii) implementing a filtering mechanism, such as using only relevant CoTs to infer answers and disregarding irrelevant ones.

## 7 Conclusion

This paper formally studies the problem of multimodal CoT. We propose Multimodal-CoT that incorporates language and vision modalities into a two-stage framework that separates rationale generation and answer inference, so answer inference can leverage better generated rationales from multimodal information. With Multimodal-CoT, our model under 1 billion parameters achieves state-of-the-art performance on the ScienceQA benchmark. Analysis shows that Multimodal-CoT has the merits of mitigating hallucination and enhancing convergence speed. Our error analysis identifies the potential to leverage more effective vision features, inject commonsense knowledge, and apply filtering mechanisms to improve CoT reasoning in future studies.

## Appendix A Extended Analysis for the Challenge of Multimodal-CoT

### A.1 Additional Examples of Misleading through Hallucinated Rationales

Based on our case studies (Section 3.2), we have observed a tendency for the baseline model to generate hallucinated rationales. Here, we present additional examples to illustrate this phenomenon, as depicted in Figure 7 .

### A.2 Two-Stage Training Performance with Different Sizes of LMs

In Section 3, we observed that the inclusion of vision features has a positive impact on the generation of more effective rationales, consequently resulting in improved answer accuracy. In addition to incorporating vision features, another approach to addressing the issue of incorrect rationales is to scale the size of the language model (LM). Figure 8 showcases the answer accuracy achieved by our two-stage training framework, both with and without the integration of vision features. Notably, when employing a larger LM, the baseline accuracy (without vision features) experiences a significant enhancement. This finding suggests that scaling the LM size could potentially alleviate the problem of incorrect rationales. However, it is crucial to acknowledge that the performance still falls considerably short of utilizing vision features. This outcome further validates the effectiveness of our Multimodal-CoT methodology across varying LM sizes.

### A.3 Discussion of the Possible Paradigms to Achieve Multimodal-CoT

As discussed in Section 1, there are two primary approaches to facilitate Multimodal-CoT reasoning: (i) prompting LLMs and (ii) fine-tuning small models. The common approach in the first approach is to unify the input from different modalities and prompt LLMs to perform reasoning ( Zhang et al., 2023a ; Lu et al., 2023 ; Liu et al., 2023 ; Alayrac et al., 2022 ; Hao et al., 2022 ; Yasunaga et al., 2022 ) . For instance, one way to achieve this is by extracting the caption of an image using a captioning model and then concatenating the caption with the original language input to feed LLMs. By doing so, visual information is conveyed to LLMs as text, effectively bridging the gap between modalities. This approach can be represented as the input-output format <image → \rightarrow caption, question + caption → \rightarrow answer>. We refer to this approach as Caption-based Reasoning (Figure 9 a). It is worth noting that the effectiveness of this approach depends on the quality of the image caption, which may be susceptible to errors introduced during the transfer from image captioning to answer inference.

In contrast, an intriguing aspect of CoT is the ability to decompose complex problems into a series of simpler problems and solve them step by step. This transformation leads to a modification of the standard format <question → \rightarrow answer> into <question → \rightarrow rationale → \rightarrow answer>. Rationales, being more likely to reflect the reasoning processes leading to the answer, play a crucial role in this paradigm. Consequently, we refer to approaches following this paradigm as CoT-based Reasoning . The nomenclature has been widely adopted in the literature ( Huang & Chang, 2022 ; Zhang et al., 2023d ; Lu et al., 2022c ) .

Our work aligns with the paradigms of CoT-based Reasoning in the context of multimodal scenarios, specifically employing the <question + image → \rightarrow rationale → \rightarrow answer> framework (Figure 9 b). This approach confers advantages on two fronts. Firstly, the Multimodal-CoT framework leverages feature-level interactions between vision and language inputs, enabling the model to gain a deeper understanding of the input information and facilitating more effective inference of answers by incorporating well-founded rationales. Our analysis has demonstrated that Multimodal-CoT offers notable benefits by mitigating hallucination and enhancing convergence, resulting in superior performance on our benchmark datasets. Secondly, the lightweight nature of Multimodal-CoT renders it compatible with resource constraints and circumvents any potential paywalls.

## Appendix B Experimental Details

### B.1 Details of Vision Features

In Section 6.2, we compared four types of vision features, ViT ( Dosovitskiy et al., 2021b ) , CLIP ( Radford et al., 2021 ) , DETR ( Carion et al., 2020 ) , and ResNet ( He et al., 2016 ) . The specific models are: (i) ViT: vit_large_patch32_384 , 8 8 8 https://github.com/rwightman/pytorch-image-models . (ii) CLIP: RN101; 9 9 9 https://github.com/jianjieluo/OpenAI-CLIP-Feature . (iii) DETR: detr_resnet101_dc5 ; 10 10 10 https://github.com/facebookresearch/detr . (iv) ResNet: we use the averaged pooled features of a pre-trained ResNet50 CNN.

Table 12 presents the dimension of the vision features (after the function VisionExtractor( ⋅ \cdot ) in Eq. 3). For ResNet-50, we repeat the pooled features of ResNet-50 to the same length as the text sequence to imitate the patch-like features, where each patch is the same as the pooled image features.

### B.2 Datasets

Our method is evaluated on the ScienceQA ( Lu et al., 2022a ) and A-OKVQA ( Schwenk et al., 2022 ) benchmark datasets.

∙ \bullet ScienceQA is a large-scale multimodal science question dataset with annotated lectures and explanations. It contains 21 k k multimodal multiple choice questions with rich domain diversity across 3 subjects, 26 topics, 127 categories, and 379 skills. The dataset is split into training, validation, and test splits with 12 k k , 4 k k , and 4 k k questions, respectively.

∙ \bullet A-OKVQA is a knowledge-based visual question answering benchmark, which has 25 k k questions requiring a broad base of commonsense and world knowledge to answer. Each question is annotated with rationales that explain why a particular answer was correct according to necessary facts or knowledge. It has 17 k k /1 k k /6 k k questions for train/val/test.

For ScienceQA, our model is evaluated on the test set. For A-OKVQA, our model is evaluated on the validation set as the test set is hidden.

### B.3 Implementation Details of Multimodal-CoT

As the Multimodal-CoT task requires generating the reasoning chains and leveraging the vision features, we adopt the T5 encoder-decoder architecture ( Raffel et al., 2020 ) under Base (200M) and large (700M) settings in our framework. We apply FLAN-Alpaca to initialize our model weights. 11 11 11 https://github.com/declare-lab/flan-alpaca . We will show that Multimodal-CoT is generally effective with other backbone LMs, such as UnifiedQA ( Khashabi et al., 2020 ) and FLAN-T5 ( Chung et al., 2022 ) (Section 6.1). The vision features are obtained by the frozen ViT-large encoder ( Dosovitskiy et al., 2021b ) . Since using image captions can slightly improve model performance, as shown in Section 3.3, we append the image captions to the context following Lu et al. (2022a) . The captions are generated by InstructBLIP ( Dai et al., 2023 ) . We fine-tune the models up to 20 epochs, with a learning rate selected in {5e-5, 8e-5}. The maximum input sequence lengths for rationale generation and answer inference are 512 and 64, respectively. The batch size is 8. Our experiments are run on 8 NVIDIA Tesla V100 32G GPUs.

## Appendix C Further Analysis

### C.1 Examples of Rationale Generation with Large Models

A recent flame is to leverage large language models or large vision-language models to generate reasoning chains for multimodal question answering problems ( Zhang et al., 2023a ; Lu et al., 2023 ; Liu et al., 2023 ; Alayrac et al., 2022 ; Hao et al., 2022 ; Yasunaga et al., 2022 ) . We are interested in whether we can use large models to generate the rationales for Multimodal-CoT; thus breaking the need for datasets with human-annotated rationales. During the first-stage training of Multimodal-CoT, our target rationales are based on human annotation in the benchmark datasets. Now, we replace the target rationales with those generated by an LLM or a vision-language model. Concretely, we feed the questions with images (IMG) and the question without images (TXT) to InstructBLIP ( Dai et al., 2023 ) (Figure 10 a) and ChatGPT (Figure 10 b) for zero-shot inference, respectively. Then, we use the generated pseudo-rationales as the target rationales for training instead of relying on the human annotation of reasoning chains.

### C.2 Detailed Results of Multimodal-CoT on Different Backbone Models

To test the generality of the benefits of our approach to other backbone models, we alter the underlying LMs to other variants of different types. As detailed results shown in Table 13 , our approach is generally effective for the widely used backbone models.

## Appendix D Examples of Case Studies

To gain deeper insights into the behavior of Multimodal-CoT and facilitate future research, we manually analyzed randomly selected examples generated by our approach. The categorization results are illustrated in Figure 11 . We examined 50 samples that yielded incorrect answers and categorized them accordingly.

The second error type is logical mistakes, constituting 14% of the errors, which involve comparison mistakes (Figure 13 a) and contradictions in the reasoning process (Figure 13 b).

Additionally, we have observed cases where incorrect answers are provided despite the CoT being either empty or correct, amounting to 6% of the errors. The CoT in these cases may not necessarily influence the final answer (Figure 14 ).

## Instructions for reporting errors

We are continuing to improve HTML versions of papers, and your feedback helps enhance accessibility and mobile support. To report errors in the HTML that will help us improve conversion and rendering, choose any of the methods listed below:

Click the "Report Issue" ( ) button, located in the page header.

Tip: You can select the relevant text first, to include it in your report.

Our team has already identified the following issues . We appreciate your time reviewing and reporting rendering errors we may not have found yet. Your efforts will help us improve the HTML versions for all readers, because disability should not be a barrier to accessing research. Thank you for your continued support in championing open access for all.

Have a free development cycle? Help support accessibility at arXiv! Our collaborators at LaTeXML maintain a list of packages that need conversion , and welcome developer contributions .