Language Is Not All You Need: Aligning Perception with Language Models — part 2/3

Cleaned-text code points [15833, 31674). Bibliography omitted here; source.md includes it.

We combine Unnatural Instructions [ 22 ] and FLANv2 [ 30 ] as our instruction dataset. Unnatural Instructions is a dataset that was created by using a large language model to generate instructions for various natural language processing tasks. It has 68,478 instruction-input-output triplets in its core dataset. FLANv2 is a collection of datasets that cover diverse types of language understanding tasks, such as reading comprehension, commonsense reasoning, and closed-book question answering. We randomly select 54k examples of instructions from FLANv2 to augment our instruction dataset. Details of the training hyperparameter settings are described in Appendix A.2 .

## 4 Evaluation

MLLMs can handle both language tasks and perception-intensive tasks. We evaluate Kosmos-1 on various types of tasks as follows:

• Language tasks – Language understanding

– Language generation

– OCR-free text classification

• Cross-modal transfer – Commonsense reasoning

• Nonverbal reasoning – IQ Test (Raven’s Progressive Matrices)

• Perception-language tasks – Image captioning

– Visual question answering

– Web page question answering

• Vision tasks – Zero-shot image classification

– Zero-shot image classification with descriptions

### 4.1 Perception-Language Tasks

We evaluate the perception-language capability of Kosmos-1 under vision-language settings. Specifically, we conduct zero-shot and few-shot experiments on two widely used tasks, including image captioning and visual question answering. Image captioning involves generating a natural language description of an image, while visual question answering aims to answer a natural language question with respect to an image.

#### 4.1.1 Evaluation Setup

We evaluate the caption generation on MS COCO Caption [ 32 ] , and Flickr30k [ 60 ] . We use the test set of COCO Karpathy split [ 24 ] , which re-partitions the train2014 and val2014 images [ 32 ] into 113,287, 5,000, and 5,000 for the training set, validation set, and test set, respectively. We conduct an evaluation on Flickr30k’s Karpathy split test set. The image resolution is 224 × \times 224. We use beam search to generate the captions, and the beam size is 5. In the few-shot settings, we randomly sample demonstrations from the training set. We use COCOEvalCap 3 3 3 https://github.com/salaniz/pycocoevalcap to compute CIDEr [ 49 ] and SPICE [ 2 ] scores as the evaluation metrics. We prompt Kosmos-1 with “An image of” for zero-shot and few-shot caption generation experiments.

For visual question-answering tasks, we evaluate zero-shot and few-shot results on test-dev set of VQAv2 [ 18 ] and test-dev set of VizWiz [ 19 ] , respectively. The resolution of images is 224 × \times 224. We use greedy search for the decoding. We follow the normalization rules of the VQAv2 evaluation code 4 4 4 https://github.com/GT-Vision-Lab/VQA when computing the VQA accuracy. We evaluate the performance of VQA in an open-ended setting that Kosmos-1 generates answers and stops at the </s> (“end of sequence”) token. The prompt is “Question: { question } Answer: { answer } ” for visual question answering tasks.

#### 4.1.2 Results

##### Image Captioning

Table 2 shows the zero-shot captioning performance on COCO Karpathy test split and Flickr30k test set. Kosmos-1 achieves remarkable results in zero-shot setting on two image captioning datasets. Specifically, our model achieves a CIDEr score of 67.1 on the Flickr30k dataset, compared to 60.6 and 61.5 for the Flamingo-3B and Flamingo-9B models, respectively. Notably, our model is able to accomplish this feat with a smaller size of 1.6B, compared to Flamingo models. This demonstrates our model’s superiority in zero-shot image captioning.

Table 3 reports the results of the few-shot ( k = 2 , 4 , 8 k=2,4,8 ) settings. The overall performance improves as the number of shots increases from two to four. The trends are consistent across the two datasets. Moreover, the few-shot results outperform zero-shot captioning in Table 2 .

##### Visual Question Answering

Table 4 reports the zero-shot visual question answering results on VQAv2 and VizWiz. We show that Kosmos-1 can better handle the diversity and complexity of the VizWiz dataset. Kosmos-1 achieves higher accuracy and robustness than Flamingo-3B and Flamingo-9B models. In addition, our model is competitive with Flamingo on the VQAv2 dataset.

Table 5 shows the few-shot performance on visual question answering tasks. Kosmos-1 outperforms other models in few-shot ( k = 2 , 4 k=2,4 ) settings on the VizWiz dataset. We also observe a positive correlation between the number of shots and the quality of the results on the VizWiz dataset. Moreover, the few-shot results are better than the zero-shot numbers as reported in Table 4 .

### 4.2 IQ Test: Nonverbal Reasoning

Raven’s Progressive Matrices [ 10 , 23 ] is one of the most common tests to evaluate nonverbal reasoning. The capability of nonverbal reasoning is typically a reflection of an individual’s intelligence quotient (IQ). Figure 4 shows an example. Given eight images presented in a 3 × 3 3\times 3 matrix, the task is to identify the following element from six similar candidates.

The models need to conduct zero-shot nonverbal reasoning without explicitly fine-tuning. The Raven IQ test is analogous to in-context learning of language models, where the difference is whether the context is nonverbal or verbal. In order to infer the answers, the models have to recognize abstract concepts and identify the underlying patterns of given images. So the IQ task is a good testbed to benchmark the nonverbal in-context learning capability.

#### 4.2.1 Evaluation Setup

To evaluate the Kosmos-1 on zero-shot nonverbal reasoning, we construct a dataset of the Raven IQ test. It consists of 50 50 examples collected from different websites 5 5 5 https://en.testometrika.com/intellectual/iq-test/ 6 6 6 https://en.testometrika.com/intellectual/iq-test-for-kids-7-to-16-year-old/ 7 7 7 https://iqpro.org/ 8 8 8 https://iqhaven.com/matrix-g . Each example has three (i.e., 2 × 2 2\times 2 matrix), four, or eight (i.e., 3 × 3 3\times 3 matrix) given images. The goal is to predict the next one. Each instance has six candidate images with a unique correct completion. We measure accuracy scores to evaluate the models. The evaluation dataset is available at https://aka.ms/kosmos-iq50 .

Figure 4 illustrates how to evaluate Kosmos-1 on the Raven IQ test. The matrix-style images are flattened and fed into the models one-by-one. To enable the model to better understand the desired task, we also use a textual instruction “Here are three/four/eight images:” , “The following image is:” , and “Is it correct?” for conditioning. We append each possible candidate to the context separately and compare the probability that the model outputs “Yes” in a close-ended setting. The candidate that yields the largest probability is regarded as the prediction.

#### 4.2.2 Results

Table 6 shows the evaluation results on the IQ test dataset. Both Kosmos-1 with and without language-only instruction tuning achieve 5.3% and 9.3% improvement respectively over the random baseline. The results indicate that Kosmos-1 is able to perceive abstract conceptual patterns in a nonverbal context, and then deduce the following element across multiple choices. To the best of our knowledge, it is the first time that a model can perform such zero-shot Raven IQ tests. Although there is still a large performance gap between the current model and the average level of adults, Kosmos-1 demonstrates the potential of MLLMs to perform zero-shot nonverbal reasoning by aligning perception with language models.

### 4.3 OCR-Free Language Understanding

OCR-free language understanding is a task that focuses on understanding text and images without relying on Optical Character Recognition (OCR). For example, during the Rendered SST-2 task, sentences from the Stanford Sentiment Treebank [ 46 ] dataset are rendered as images. The model is asked to predict the sentiment of the text within the images. The task evaluates a model’s ability to read and comprehend the meaning of words and sentences directly from the images.

#### 4.3.1 Evaluation Setup

We evaluate OCR-free language understanding on the Rendered SST-2 [ 38 ] test set and HatefulMemes [ 25 ] validation set. We use accuracy as the metric for the Rendered SST-2 and report ROC AUC for the HatefulMemes dataset. We use the prompt “Question: what is the sentiment of the opinion? Answer: { answer } ” , where the answer is either positive or negative for the Rendered SST-2. For the HatefulMemes task, the prompt is “Question: does this picture contain real hate speech? Answer: { answer } ” , where the answer is either yes or no.

#### 4.3.2 Results

As shown in Table 7 , Kosmos-1 achieves a ROC AUC of 63.9% for the HatefulMemes validation set and a test accuracy of 67.1% for the Rendered SST-2 test set. It outperforms CLIP ViT-L and Flamingo-9B, which achieve AUCs of 63.3% and 57.0% on the HatefulMemes task. Note that Flamingo explicitly provides OCR text into the prompt, while Kosmos-1 does not access any external tools or resources. This indicates that Kosmos-1 has built-in abilities to read and comprehend the text in the rendered images.

### 4.4 Web Page Question Answering

Web page question answering aims at finding answers to questions from web pages. It requires the model to comprehend both the semantics and the structure of texts. The structure of the web page (such as tables, lists, and HTML layout) plays a key role in how the information is arranged and displayed. The task can help us evaluate our model’s ability to understand the semantics and the structure of web pages.

#### 4.4.1 Evaluation Setup

We compare the performance on the Web-based Structural Reading Comprehension (WebSRC) dataset [ 14 ] . For comparisons, we train a language model ( LLM ) on the same text corpora with the same training setup as in Kosmos-1 . The LLM takes the text extracted from the web page as input. Its template of the prompt is “Given the context below from web page, extract the answer from the given text like this: Qusestion: Who is the publisher of this book? Answer: Penguin Books Ltd. Context: { WebText } Q: { question } A: { answer } ” , where the { WebText } presents the text extracted from the web page. Besides using the same prompt, Kosmos-1 prepends the image before the prompt. Two example images from WebSRC are shown in Appendix C.3 . Following the original paper [ 14 ] , we use exact match (EM) and F1 scores as our evaluation metrics.

#### 4.4.2 Results

The experimental results are summarized in Table 8 . We observe that Kosmos-1 outperforms the LLM , indicating that Kosmos-1 can benefit from the layout and style information of web pages in images. In addition, we evaluate the performance of Kosmos-1 without the extracted text in the prompt. It shows that extracted text has a contribution of +12.0/20.7 EM/F1 to Kosmos-1 , indicating that the benefit from modeling images does not sacrifice its language abilities.

### 4.5 Multimodal Chain-of-Thought Prompting

Chain-of-thought prompting [ 57 ] allows large language models to generate a series of reasoning steps and decompose a multi-step problem into intermediate steps, which can significantly improve the performance in complex tasks. Motivated by chain-of-thought prompting, we investigate a multimodal chain-of-thought prompting using Kosmos-1 . As illustrated in Figure 5 , we break down perception-language tasks into two steps. In the first stage, given an image, we use a prompt to guide the model to generate a rationale. The model is then fed the rationale and a task-aware prompt to produce the final results.

#### 4.5.1 Evaluation Setup

We evaluate the ability of multimodal chain-of-thought prompting on the Rendered SST-2. We use the prompt “Introduce this picture in detail:” to generate the content in the picture as the rationale. Then, we use the prompt “ { rationale } Question: what is the sentiment of the opinion? Answer: { answer } ” to predict the sentiment, where the answer is either positive or negative.

#### 4.5.2 Results

We conduct experiments to evaluate the performance of the multimodal chain-of-thought prompting. Table 9 shows that multimodal chain-of-thought prompting achieves a score of 72.9, which is 5.8 points higher than the standard prompting. By generating intermediate content, the model can recognize the text in the images and infer the sentiment of the sentences more correctly.

### 4.6 Zero-Shot Image Classification

We report the zero-shot image classification performance on ImageNet [ 15 ] . Image classification comprehends an entire image as a whole and aims to assign a label to the image. We map each label to its category name in natural language. The model is prompted to predict the category name to perform zero-shot image classification.

#### 4.6.1 Evaluation Setup

Given an input image, we concatenate the image with the prompt “The photo of the” . The input is then fed into the model to obtain the category name of the image. We evaluate the model on ImageNet [ 15 ] , which contains 1.28M training images and 50k validation images in 1k object categories. The prediction is classified as correct if it is exactly the same as the ground-truth category name. The image resolution used for evaluation is 224 × \times 224. We use beam search to generate the category names and the beam size is 2.

#### 4.6.2 Results

As shown in Table 10 , we report zero-shot results in both constrained and unconstrained settings. The difference between the two settings is whether we use the 1k object category names to constrain the decoding. Kosmos-1 significantly outperforms GIT [ 58 ] by 4.6% under the constrained setting and 2.1% under the unconstrained setting.

### 4.7 Zero-Shot Image Classification with Descriptions

The standard approach of image classification as above is to prompt the model for the specific name of the object depicted in the image. However, there are also some classification rules customized for different users and scenarios, such as the refined classification of complex animal subspecies. We can utilize natural language descriptions to guide Kosmos-1 to distinguish images in the zero-shot setting, which makes the decision process more interpretable.

#### 4.7.1 Evaluation Setup

Following CUB [ 51 ] , we construct a bird classification dataset that contains images and natural-language descriptions of categories. The dataset has three groups of binary image classification. Each group contains two animal categories with similar appearances. Our goal is to classify images given the categories’ descriptions. Table 11 presents the data samples. The first group is from [ 51 ] , while the other two groups are collected from the website. Each category contains twenty images.

The evaluation procedure is illustrated in Figure 6 . For the zero-shot setting, we provide detailed descriptions of two specific categories and use the template “Question:what is the name of { general category } in the picture? Answer:” to prompt the model for the specific category name in an open-ended manner. To evaluate the effect of providing verbal descriptions in context, we also implement a zero-shot baseline without prompting descriptions. Instead, we provide the corresponding specific names in the prompt.

#### 4.7.2 Results

The evaluation results are shown in Table 12 . We observe that providing descriptions in context can significantly improve the accuracy of image classification. The consistent improvements indicate that Kosmos-1 can perceive the intentions of instructions and well align the concepts in language modality with visual features in vision modality.

### 4.8 Language Tasks