Language Is Not All You Need: Aligning Perception with Language Models — part 3/3

Cleaned-text code points [31674, 41312). Bibliography omitted here; source.md includes it.

The models are evaluated on the language tasks given task instructions (i.e., zero-shot) or several demonstration examples (i.e., few-shot). Text inputs are directly fed into the models as in vanilla language models.

#### 4.8.1 Evaluation Setup

We train a language model ( LLM ) baseline with the same text corpora and training setup. We evaluate Kosmos-1 and the LLM baseline on eight language tasks, including cloze and completion tasks (i.e, StoryCloze, HellaSwag), Winograd-style tasks (i.e, Winograd, Winogrande), commonsense reasoning (i.e, PIQA), and three datasets BoolQ, CB, and COPA from the SuperGLUE benchmark [ 56 ] . The detailed descriptions of these datasets are provided in Appendix C.2 . We conduct experiments under zero-shot and few-shot settings. We evaluate each test example by randomly sampling examples from the training set as demonstrations. We set the number of shots to 0, 1, and 4 in our experiments.

#### 4.8.2 Results

Table 13 presents the in-context learning performance of language tasks. Kosmos-1 achieves comparable or even better performance in cloze completion and commonsense reasoning tasks when compared to LLM . In terms of the average result across all these datasets, LLM performs better in zero-shot and one-shot settings, whereas our model performs better in few-shot ( k = 4 k=4 ) settings. The results indicate that Kosmos-1 also handles language-only tasks well and achieves favorable performance across datasets. In addition, Section 4.9.2 shows that MLLMs learn better visual commonsense knowledge compared with LLMs.

### 4.9 Cross-modal Transfer

Cross-modal transferability allows a model to learn from one modality (such as text, image, audio, etc.) and transfer the knowledge to the other modalities. This skill can enable a model to perform various tasks across different modalities. In this part, we evaluate the cross-model transferability of Kosmos-1 on several benchmarks.

#### 4.9.1 Transfer from Language to Multimodal: Language-Only Instruction Tuning

To evaluate the effect of language-only instruction tuning, we conduct an ablation study using four datasets: COCO, Flickr30k, VQAv2, and VizWiz. These datasets consist of image captioning and visual questions anwsering. The evaluation metrics are: CIDEr scores for COCO/Flickr30k and VQA accuracy for VQAv2/VizWiz.

Table 14 shows the experimental results. Language-only instruction tuning boosts our model’s performance by 1.9 points on Flickr30k, 4.3 points on VQAv2, and 1.3 points on VizWiz. Our experiments show that language-only instruction tuning can significantly improve the model’s instruction-following capabilities across modalities. The results also indicate that our model can transfer the instruction-following capability from language to other modalities.

#### 4.9.2 Transfer from Multimodal to Language: Visual Commonsense Reasoning

Visual commonsense reasoning tasks require an understanding of the properties of everyday objects in the real world, such as color, size, and shape. These tasks are challenging for language models because they may require more information about object properties than what is available in texts. To investigate the visual commonsense capabilities, we compare the zero-shot performance of Kosmos-1 and LLM on visual commonsense reasoning tasks.

##### Evaluation Setup

We compare Kosmos-1 and the LLM baseline on three object commonsense reasoning datasets, RelativeSize [ 5 ] , MemoryColor [ 36 ] and ColorTerms [ 4 ] datasets. Table 15 shows some examples of object size and color reasoning tasks. RelativeSize contains 486 object pairs from 41 physical objects. The model is required to predict the size relation between two objects in a binary question-answering format with “Yes”/“No” answers. MemoryColor and ColorTerms require the model to predict the color of objects from a set of 11 color labels in a multiple-choice format. We use only text as our input and do not include any images. We measure the accuracy of our model on these three datasets.

##### Results

Table 16 presents the zero-shot performance of Kosmos-1 and LLM on visual commonsense reasoning tasks. Kosmos-1 significantly outperforms LLM by 1.5% on RelativeSize , 14.7% on MemoryColor , and 9.7% on ColorTerms dataset. The consistent improvements indicate that Kosmos-1 benefits from the visual knowledge to complete the corresponding visual commonsense reasoning. The reason for Kosmos-1 ’s superior performance is that it has modality transferability, which enables the model to transfer visual knowledge to language tasks. On the contrary, LLM has to rely on textual knowledge and clues to answer visual commonsense questions, which limits its ability to reason about object properties.

## 5 Conclusion

In this work, we introduce Kosmos-1 , a multimodal large language model that can perceive general modalities, follow instructions, and perform in-context learning. The models trained on web-scale multimodal corpora achieve promising results across a wide range of language tasks and multimodal tasks. We show that going from LLMs to MLLMs enables new capabilities and opportunities. In the future, we would like to scale up Kosmos-1 in terms of model size [ 35 , 55 , 9 ] , and integrate the speech [ 52 ] capability into Kosmos-1 . In addition, Kosmos-1 can be used as a unified interface for multimodal learning, e.g., enabling using instructions and examples to control text-to-image generation.

## Appendix A Hyperparameters

### A.1 Training

We report the detailed model hyperparameter settings of Kosmos-1 in Table 17 and training hyperparameters in Table 18 .

### A.2 Language-Only Instruction Tuning

The detailed instruction tuning hyperparameters are listed in Table 19 .

## Appendix B Datasets

### B.1 Pretraning

#### B.1.1 Text Corpora

Kosmos-1 is trained on The Pile [ 17 ] and Common Crawl. The Pile is an 800 GB English text corpus combining 22 diverse sources. We select a subset with seven sources from The Pile. Common Crawl is also included in training corpora. Common Crawl takes snapshots of the web, which contains massive amounts of language data. Table 20 provides a full overview of the language datasets that were used in the training of Kosmos-1 model. These data sources can be divided into the following three categories: • Academic : NIH Exporter

• Internet : Pile-CC, OpenWebText2, Wikipedia (English), CC-2020-50, CC-2021-04, Realnews

• Prose : BookCorpus2, Books3, Gutenberg [ 39 ] , CC-Stories

#### B.1.2 Image-Caption Pairs

Kosmos-1 is trained on image-caption pairs constructed from several datasets, including English LAION-2B [ 41 ] , LAION-400M [ 47 ] , COYO-700M [ 7 ] and Conceptual Captions [ 42 , 13 ] . LAION-2B, LAION-400M, and COYO-700M datasets are extracted by parsing out image URLs and alt-texts of web pages from the Common Crawl web data. LAION-2B contains about 2B English image-caption pairs, LAION-400M consists of 400M English image-caption pairs, and COYO-700M has 700M English image-caption pairs. Conceptual Captions contains 15M English image-caption pairs and consists of two datasets: CC3M and CC12M, which are also collected from internet webpages using a Flume pipeline. For Conceptual Captions, we discard pairs whose captions contain special tags such as “<PERSON>”.

#### B.1.3 Interleaved Data

We collect a large corpus of 2 billion web pages from the snapshots of common crawls. To ensure quality and relevance, we apply several filtering criteria. First, we discard any pages that are not written in English. Second, we discard any pages that do not have images interspersed in the text. Third, we discard any images that have a resolution lower than 64 by 64 pixels or that are single-colored. Fourth, we discard any text that is not meaningful or coherent, such as spam or gibberish. We use some heuristics to identify and remove gibberish text containing emoji symbols, hashtags, and URL links. After applying these filters, we end up with about 71 million documents for training.

### B.2 Data Format

The training data is organized in the format as follows:

## Appendix C Evaluation

### C.1 Input Format Used for Perception-Language Tasks

Figure 7 shows how we conduct zero-shot and few-shot evaluations on perception-language tasks.

### C.2 Language Tasks

We conduct experiments on language tasks in four categories: • Cloze and completion tasks: StoryCloze [ 34 ] , HellaSwag [ 61 ]

• Winograd-style tasks: Winograd [ 29 ] , Winogrande [ 40 ]

• Commonsense reasoning: PIQA [ 8 ]

• Three datasets from SuperGLUE benchmark [ 56 ] : BoolQ [ 11 ] , CB [ 16 ] , COPA [ 37 ]

### C.3 WebSRC Task Examples

3 59 48 31 27

## Instructions for reporting errors

We are continuing to improve HTML versions of papers, and your feedback helps enhance accessibility and mobile support. To report errors in the HTML that will help us improve conversion and rendering, choose any of the methods listed below:

Click the "Report Issue" ( ) button, located in the page header.

Tip: You can select the relevant text first, to include it in your report.

Our team has already identified the following issues . We appreciate your time reviewing and reporting rendering errors we may not have found yet. Your efforts will help us improve the HTML versions for all readers, because disability should not be a barrier to accessing research. Thank you for your continued support in championing open access for all.

Have a free development cycle? Help support accessibility at arXiv! Our collaborators at LaTeXML maintain a list of packages that need conversion , and welcome developer contributions .