Multimodal Chain-of-Thought Reasoning in Language Models — part 2/3
Cleaned-text code points [15995, 31801). Bibliography omitted here; source.md includes it.
With those effective rationales, the phenomenon of hallucination is mitigated — 60.7% hallucination mistakes in Section 3.2 have been corrected (Figure 3 (b)), as an example shown in Figure 2 (right part). 5 5 5 The left mistakes are mainly about map understanding, requiring extra commonsense signals (Section 6.7 ). The analysis so far compellingly shows that vision features are indeed beneficial for generating effective rationales and contributing to accurate answer inference. As the two-stage method achieves better performance than one-stage methods, we choose the two-stage method in our Multimodal-CoT framework.
## 4 Multimodal-CoT
In light of the discussions in Section 3 , we propose Multimodal-CoT. The key motivation is the anticipation that the answer inference can leverage better generated rationales that are based on multimodal information. In this section, we will overview the procedure of the framework and elaborate on the technical design of the model architecture.
### 4.1 Framework Overview
Multimodal-CoT consists of two operation stages: (i) rationale generation and (ii) answer inference. Both stages share the same model structure but differ in the input X X and output Y Y . The overall procedure is illustrated in Figure 4 . We will take vision-language as an example to show how Multimodal-CoT works.
In the rationale generation stage, we feed the model with X = { X language 1 , X vision } X=\{X_{\textrm{language}}^{1},X_{\textrm{vision}}\} where X language 1 X_{\textrm{language}}^{1} represents the language input in the first stage and X vision X_{\textrm{vision}} represents the vision input, i.e., the image. For example, X X can be instantiated as a concatenation of question, context, and options of a multiple choice reasoning problem ( Lu et al., 2022a ) as shown in Figure 4 . The goal is to learn a rationale generation model R = F ( X ) {R}=F(X) where R {R} is the rationale.
In the answer inference stage, the rationale R R is appended to the original language input X language 1 X_{\textrm{language}}^{1} to construct the language input in the second stage, X language 2 = X language 1 ∘ R X_{\textrm{language}}^{2}=X_{\textrm{language}}^{1}\circ{R} where ∘ \circ denotes concatenation. Then, we feed the updated input X ′ = { X language 2 , X vision } X^{\prime}=\{X_{\textrm{language}}^{2},X_{\textrm{vision}}\} to the answer inference model to infer the final answer A = F ( X ′ ) A=F(X^{\prime}) .
In both stages, we train two models with the same architecture independently. They take the annotated elements (e.g., X → R X\rightarrow R , X R → A XR\rightarrow A , respectively) from the training set for supervised learning. During inference, given X X , the rationales for the test sets are generated using the model trained in the first stage; they are used in the second stage for answer inference.
### 4.2 Model Architecture
Given language input X language ∈ { X language 1 , X language 2 } X_{\textrm{language}}\in\{X_{\textrm{language}}^{1},X_{\textrm{language}}^{2}\} and vision input X vision X_{\textrm{vision}} , we compute the probability of generating target text Y Y (either the rationale or the answer in Figure 4 ) of length N N by p ( Y | X language , X vision ) = ∏ i = 1 N p θ ( Y i ∣ X language , X vision , Y < i ) , p(Y|X_{\textrm{language}},X_{\textrm{vision}})=\prod_{i=1}^{N}p_{\theta}\left(Y_{i}\mid X_{\textrm{language}},X_{\textrm{vision}},Y_{<i}\right), (1) where p θ ( Y i ∣ X language , X vision , Y < i ) p_{\theta}\left(Y_{i}\mid X_{\textrm{language}},X_{\textrm{vision}},Y_{<i}\right) is implemented with a Transformer-based network ( Vaswani et al., 2017 ) . The network has three major procedures: encoding, interaction, and decoding. Specifically, we feed the language text into a Transformer encoder to obtain a textual representation, which is interacted and fused with the vision representation before being fed into the Transformer decoder.
#### Encoding
The model F ( X ) F(X) takes both the language and vision inputs and obtains the text representation H language H_{\textrm{language}} and the image feature H vision H_{\textrm{vision}} by the following functions: H language \displaystyle H_{\textrm{language}} = \displaystyle= LanguageEncoder ( X language ) , \displaystyle\textrm{LanguageEncoder}(X_{\textrm{language}}), (2) H vision \displaystyle H_{\textrm{vision}} = \displaystyle= W h ⋅ VisionExtractor ( X vision ) , \displaystyle W_{h}\cdot\textrm{VisionExtractor}(X_{\textrm{vision}}), (3) where LanguageEncoder( ⋅ \cdot ) is implemented as a Transformer model. We use the hidden states of the last layer in the Transformer encoder as the language representation H language ∈ ℝ n × d H_{\textrm{language}}\in\mathbb{R}^{n\times d} where n n denotes the length of the language input, and d d is the hidden dimension. Meanwhile, VisionExtractor( ⋅ \cdot ) is used to vectorize the input image into vision features. Inspired by the recent success of Vision Transformers ( Dosovitskiy et al., 2021a ) , we fetch the patch-level features by frozen vision extraction models, such as ViT ( Dosovitskiy et al., 2021b ) . After obtaining the patch-level vision features, we apply a learnable projection matrix W h W_{h} to convert the shape of VisionExtractor ( X vision ) \textrm{VisionExtractor}(X_{\textrm{vision}}) into that of H language H_{\textrm{language}} ; thus we have H vision ∈ ℝ m × d H_{\textrm{vision}}\in\mathbb{R}^{m\times d} where m m is the number of patches.
Note that our approach is general to both scenarios with or without image context. For the questions without associated images, we use all-zero vectors as the “blank features” with the same shape as the normal image features to tell the model to ignore them.
#### Interaction
After obtaining language and vision representations, we use a single-head attention network to correlate text tokens with image patches, where the query ( Q {Q} ), key ( K {K} ) and value ( V {V} ) are H language H_{\textrm{language}} , H vision H_{\textrm{vision}} and H vision H_{\textrm{vision}} , respectively. The attention output H vision attn ∈ ℝ n × d H_{\textrm{vision}}^{\textrm{attn}}\in\mathbb{R}^{n\times d} is defined as: H vision attn = Softmax ( Q K ⊤ d k ) V , H_{\textrm{vision}}^{\textrm{attn}}=\textrm{Softmax}(\frac{{Q}{{K}^{\top}}}{\sqrt{d_{k}}}){V}, where d k d_{k} is the same as the dimension of H language H_{\textrm{language}} because a single head is used.
Then, we apply the gated fusion mechanism ( Zhang et al., 2020 ; Wu et al., 2021 ; Li et al., 2022a ) to fuse H language H_{\textrm{language}} and H vision H_{\textrm{vision}} . The fused output H fuse ∈ ℝ n × d H_{\textrm{fuse}}\in\mathbb{R}^{n\times d} is obtained by: λ \displaystyle\lambda = \displaystyle= Sigmoid ( W l H language + W v H vision attn ) , \displaystyle\textrm{Sigmoid}(W_{l}{H_{\textrm{language}}}+W_{v}{H_{\textrm{vision}}^{\textrm{attn}}}), (4) H fuse \displaystyle H_{\textrm{fuse}} = \displaystyle= ( 1 − λ ) ⋅ H language + λ ⋅ H vision attn , \displaystyle(1-\lambda)\cdot H_{\textrm{language}}+\lambda\cdot H_{\textrm{vision}}^{\textrm{attn}}, (5) where W l W_{l} and W v W_{v} are learnable parameters.
#### Decoding
Finally, the fused output H fuse H_{\textrm{fuse}} is fed into the Transformer decoder to predict the target Y Y .
## 5 Experiments
This section will present the benchmark dataset, the implementation of our technique, and the baselines for comparisons. Then, we will report our main results and findings.
### 5.1 Dataset
Our method is evaluated on the ScienceQA ( Lu et al., 2022a ) and A-OKVQA ( Schwenk et al., 2022 ) benchmark datasets. We choose those datasets because they are latest multimodal reasoning benchmarks with annotated reasoning chains. ScienceQA is a large-scale multimoda science question dataset with annotated lectures and explanations. It contains 21 k k multimodal multiple choice questions with rich domain diversity across 3 subjects, 26 topics, 127 categories, and 379 skills. There are 12 k k , 4 k k , and 4 k k questions in the training, validation, and test splits, respectively. A-OKVQA is a knowledge-based visual question answering benchmark, which has 25 k k questions requiring a broad base of commonsense and world knowledge to answer. It has 17 k k /1 k k /6 k k questions for train/val/test. As A-OKVQA provides multiple-choice and direct answer evaluation settings, we use the multiple-choice setting to keep consistency with ScienceQA.
### 5.2 Implementation
The following part presents the experimental settings of Multimodal-CoT and the baseline methods.
#### Experimental Settings
We adopt the T5 encoder-decoder architecture ( Raffel et al., 2020 ) under Base (200M) and large (700M) settings in our framework. We apply FLAN-Alpaca to initialize our model weights. 6 6 6 https://github.com/declare-lab/flan-alpaca . We will show that Multimodal-CoT is generally effective with other backbone LMs, such as UnifiedQA ( Khashabi et al., 2020 ) and FLAN-T5 ( Chung et al., 2022 ) (Section 6.3 ). The vision features are obtained by the frozen ViT-large encoder ( Dosovitskiy et al., 2021b ) . We fine-tune the models up to 20 epochs, with a learning rate of 5e-5. The maximum input sequence length is 512. The batch size is 8. Our experiments are run on 8 NVIDIA Tesla V100 32G GPUs. More details are presented in Appendix B .
#### Baseline Models
We utilized three categories of methods as our baselines:
(i) Visual question answering (VQA) models, including MCAN ( Yu et al., 2019 ) , Top-Down ( Anderson et al., 2018 ) , BAN ( Kim et al., 2018 ) , DFAF ( Gao et al., 2019 ) , ViLT ( Kim et al., 2021 ) , Patch-TRM ( Lu et al., 2021 ) , and VisualBERT ( Li et al., 2019 ) . These VQA baselines take the question, context, and choices as textual input, while utilizing the image as visual input. They employ a linear classifier to predict the score distribution over the choice candidates.
(ii) LMs, including the text-to-text UnifiedQA model ( Khashabi et al., 2020 ) and few-shot learning LLMs (GPT-3.5, ChatGPT, GPT-4, and Chameleon ( Lu et al., 2023 ) ). UnifiedQA ( Khashabi et al., 2020 ) is adopted as it is the best fine-tuning model in Lu et al. (2022a) . UnifiedQA takes the textual information as the input and outputs the answer choice. The image is converted into a caption extracted by an image captioning model following Lu et al. (2022a) . UnifiedQA treats our task as a text generation problem. In Lu et al. (2022a) , it is trained to generate a target answer text, i.e., one of the candidate options. Then, the most similar option is selected as the final prediction to evaluate the question answering accuracy. For GPT-3.5 models ( Chen et al., 2020 ) , we use the text-davinci-002 and text-davinci-003 engines due to their strong performance. In addition, we also include the comparison with ChatGPT and GPT-4. The inference is based on the few-shot prompting, where two in-context examples from the training set are concatenated before the test instance. The few-shot demonstrations are the same as those in Lu et al. (2022a) .
(iii) Fine-tuned large vision-language model. We select the recently released LLaMA-Adapter ( Zhang et al., 2023a ) , LLaVA ( Liu et al., 2023 ) , and InstructBLIP ( Dai et al., 2023 ) as the competitive large vision-language baselines. For LLaMA-Adapter, the backbone model is the 7B LLaMA model fine-tuned with 52 k k self-instruct demonstrations. To adapt to our tasks, the model is further fine-tuned on the ScienceQA dataset.
### 5.3 Main Results
Table 4 shows the main results in the ScienceQA benchmark. We observe that Mutimodal-CoT Large {}_{\texttt{Large}} achieves substantial performance gains over the prior best model in publications (86.54% → \rightarrow 90.45%). The efficacy of Multimodal-CoT is further supported by the results obtained from the A-OKVQA benchmark in Table 5 .
It is worth noting that Chameleon, LLaMA-Adapter, LLaVA, and InstructBLIP are concurrent works released several months after our work. In the subsequent Section 6.2 , we will show that our method is orthogonal to those multimodal models (e.g., InstructBLIP) and can be potentially used with them together to improve generality further, i.e., scaled to scenarios where human-annotated rationales are unavailable, thereby establishing the effectiveness across diverse tasks.
Ablation study results in Table 6 show that both the integration of vision features and the two-stage framework design contribute to the overall performance.
Furthermore, we find that Multimodal-CoT demonstrates the ability to mitigate hallucination (Section 3.3 ) and improve convergence (Section 6.1 ).
## 6 Analysis
The following analysis will first show that Multimodal-CoT helps enhance convergence speed and has the feasibility of adaptation to scenarios without human-annotated rationales. Then, we investigate the general effectiveness of Multimodal-CoT with different backbone models and vision features. We will also conduct an error analysis to explore the limitations to inspire future studies. We use models under the base size for analysis unless otherwise stated.
### 6.1 Multimodality Boosts Convergence
Figure 5 shows the validation accuracy curve of the baseline and Multimodal-CoT across different training epochs. “One-stage” is based on the QCM → \rightarrow A input-output format as it achieves the best performance in Table 2 and “Two-stage” is our two-stage framework. We find that the two-stage methods achieve relatively higher accuracy at the beginning than the one-stage baselines that generate the answer directly without CoT. However, without the vision features, the two-stage baseline could not yield better results as the training goes on due to low-quality rationales (as observed in Section 3 ). In contrast, using vision features helps generate more effective rationales that contribute to better answer accuracy in our two-stage multimodal variant.
### 6.2 When Multimodal-CoT Meets Large Models
A recent flame is to leverage large language models or large vision-language models to generate reasoning chains for multimodal question answering problems ( Zhang et al., 2023a ; Lu et al., 2023 ; Liu et al., 2023 ; Alayrac et al., 2022 ; Hao et al., 2022 ; Yasunaga et al., 2022 ) . We are interested in whether we can use large models to generate the rationales for Multimodal-CoT; thus breaking the need for datasets with human-annotated rationales. During the first-stage training of Multimodal-CoT, our target rationales are based on human annotation in the benchmark datasets. Now, we replace the target rationales with those generated ones. As ScienceQA contains questions with images and without images, we leverage InstructBLIP and ChatGPT to generate the rationales for questions with paired images and questions without paired images, respectively. 7 7 7 Examples are provided in Appendix C.1 . Then, we combine both of the generated pseudo-rationales as the target rationales for training (Multimodal-CoT w/ Generation) instead of relying on the human annotation of reasoning chains (Multimodal-CoT w/ Annotation).
Table 7 shows the comparison results. We see that using the generated rationales achieves comparable performance to using human-annotated rationales for training. In addition, the performance is also much better than directly prompting those baseline models to obtain the answer (in the QCM → \rightarrow A inference format).
We see that Multimodal-CoT can work effectively with large models. The findings above compellingly show the feasibility of adaptation to scenarios without human-annotated rationales, thereby establishing the effectiveness of our approach across diverse tasks.
### 6.3 Effectiveness Across Backbones