The Llama 3 Herd of Models — part 13/16

Cleaned-text code points [186342, 202188). Bibliography omitted here; source.md includes it.

We train a vision reward model (RM) on top of the vision SFT model and the language RM. The vision encoder and the cross-attention layers are initialized from the vision SFT model and unfrozen during training, while the self-attention layers are initialized from the language RM and kept frozen. We observe that freezing the language RM part generally leads to better accuracy, especially on tasks that require the RM to judge based on its knowledge or the language quality. We adopt the same training objective as the language RM, but adding a weighted regularization term on the square of the reward logits averaged over the batch, which prevents the reward scores from drifting.

The human preference annotations in Section 7.5.3 are used to train the vision RM. We follow the same practice as language preference data (Section 4.2.1 ) to create two or three pairs with clear ranking ( edited ¿ chosen ¿ rejected ). In addition, we also synthetically augment the negative responses by perturbing the words or phrases related to the information in the image (such as numbers or visual texts). This encourages the vision RM to ground its judgement based on the actual image content.

#### 7.5.5 Direct Preference Optimization

Similar to the language model (Section 4.1.4 ), we further train the vision adapters with Direct Preference Optimization (DPO; Rafailov et al. (2023) ) using the preference data described in Section 7.5.3 . To combat the distribution shift during post-training rounds, we only keep recent batches of human preference annotations while dropping batches that are sufficiently off-policy ( e.g. , if the base pre-trained model is changed). We find that instead of always freezing the reference model, updating it in an exponential moving average (EMA) fashion every k-steps helps the model learn more from the data, resulting in better performance in human evaluations. Overall, we observed that the vision DPO model consistently performs better than its SFT starting point in human evaluations for every finetuning iteration.

#### 7.5.6 Rejection Sampling

Most available question-answer pairs only contain the final answer and lack the chain-of-thought explanation that is required to train a model that generalizes well for reasoning tasks. We use rejection sampling to generate the missing explanations for such examples and boost the model’s reasoning capabilities.

Given a question-answer pair, we generate multiple answers by sampling the finetuned model with different system prompts or temperature. Next, we compare the generated answers to the ground-truth via heuristics or an LLM judge. Finally, we retrain the model by adding the correct answers back into the finetuning data mix. We find it useful to keep multiple correct answers per question.

To ensure we only add high-quality examples back into training, we implemented the following two guardrails. First, we find that some examples contain incorrect explanations, despite the final answer being correct. We observed that this pattern occurs more frequently for questions where only a small fraction of the generated answers is correct. Therefore, we drop answers for questions where the probability of the answer being correct is below a certain threshold. Second, raters prefer some answers over others due to differences in language or style. We use the reward model to select top- K K highest-quality answers and add them back into training.

#### 7.5.7 Quality Tuning

We curate a small but highly selective SFT dataset where all samples have been rewritten and verified either by humans or our best models to meet our highest standards. We train DPO models with this data to improve response quality, calling the process Quality-Tuning (QT). We find that QT significantly improves human evaluations without affecting generalization verified by benchmarks when the QT dataset covers a wide range of tasks and proper early stopping is applied. We select checkpoints at this stage purely based on benchmarks to ensure capabilities are retained or improved.

### 7.6 Image Recognition Results

We evaluate the performance of the image understanding capabilities of Llama 3 on a range of tasks spanning natural image understanding, text understanding, charts understanding and multimodal reasoning: • MMMU ( Yue et al., 2024a ) is a challenging dataset for mulitmodal reasoning where model is expected to understand images and solve college-level problems spanning 30 different disciplines. This includes both multiple-choice and open ended questions. We evaluate our model on the validation set with 900 images, in line with other works.

• VQAv2 ( Antol et al., 2015 ) tests the ability of a model to combine image understanding, language understanding and commonsense knowlege to answer generic questions about natural images

• AI2 Diagram ( Kembhavi et al., 2016 ) evaluates models capability to parse scientific diagrams and answer questions about the same. We use the same evaluation protocol as Gemini and x.ai, and report scores using a transparent bounding box.

• ChartQA ( Masry et al., 2022 ) is a challenging benchmark for charts understanding. This requires model to visually understand different kinds of charts and answer logical questions about the charts.

• TextVQA ( Singh et al., 2019 ) is a popular benchmark dataset that requires models to read and reason about text in images to answer questions about them. This tests the OCR understanding ability of the model on natural images.

• DocVQA ( Mathew et al., 2020 ) is a benchmark dataset focused on document analysis and recognition. It contains images of a wide range of documents which evaluates a model’s ability to perform OCR understanding and reason about the contents of a document to answer questions about them.

Table 29 presents the results of our experiments. The results in the table show that our vision module attached to Llama 3 performs competitively across a wide range of image-recognition benchmarks at varying model capacities. Using the resulting Llama 3-V 405B model, we outperform GPT-4V on all benchmarks, while being slightly behind Gemini 1.5 Pro and Claude 3.5 Sonnet. Llama 3 405B appears particularly competitive on document understanding tasks.

### 7.7 Video Recognition Results

We evaluate our video adapter for Llama 3 on three benchmarks: • PerceptionTest ( Pătrăucean et al., 2023 ) evaluates the model’s ability to answer temporal reasoning questions focusing on skills (memory, abstraction, physics, semantics) and different types of reasoning (descriptive, explanatory, predictive, counterfactual). It consists of 11.6 ​ K 11.6K test QA pairs, each with an on-average 23 ​ s 23s long video, filmed by 100 100 participants worldwide to show perceptually interesting tasks. We focus on the multiple-choice question answering task, where each question is paired with three possible options. We report performance on the held-out test split which is accessed by submitting our predictions to an online challenge server. 16 16 16 See https://eval.ai/web/challenges/challenge-page/2091/overview .

• NExT-QA ( Xiao et al., 2021 ) is another temporal and causal reasoning benchmark, with a focus on open-ended question answering. It consists of 1 ​ K 1K test videos each on-average 44 ​ s 44s in length, paired with 9 ​ K 9K questions. The evaluation is performed by comparing the model’s responses with the ground truth answer using Wu-Palmer Similarity (WUPS) ( Wu and Palmer, 1994 ) . 17 17 17 See https://github.com/doc-doc/NExT-OE .

• TVQA ( Lei et al., 2018 ) evaluates the model’s ability to perform compositional reasoning, requiring spatiotemporal localization of relevant moments, recognition of visual concepts, and joint reasoning with subtitle-based dialogue. This dataset, being derived from popular TV shows, additionally tests for the model’s ability to leverage its outside-knowledge of those TV shows in answering the questions. It consists of over 15 ​ K 15K validation QA pairs, with each corresponding video clip being on-average 76 ​ s 76s in length. It also follows a multiple-choice format with five options for each question, and we report performance on the validation set following prior work ( OpenAI, 2023b ) .

• ActivityNet-QA ( Yu et al., 2019 ) evaluates the model’s ability to reason over long video clips to understand actions, spatial relations, temporal relations, counting, etc. It consists of 8 ​ K 8K test QA pairs from 800 800 videos, each on-average 3 3 minutes long. For evaluation, we follow the protocol from prior work ( Google, 2023 ; Lin et al., 2023 ; Maaz et al., 2024 ) , where the model generates short one-word or one-phrase answers, and the correctness of the output is evaluated using the GPT-3.5 API which compares it to the ground truth answer. We report the average accuracy as evaluated by the API.

When performing inference, we uniformly sample frames from the full video clip and pass those frames into the model with a short text prompt. Since most of our benchmarks involve answering multiple-choice questions, we use the following prompt: Select the correct answer from the following options: {question}. Answer with the correct option letter and nothing else . For benchmarks that require producing a short answer ( e.g. , ActivityNet-QA and NExT-QA), we use the following prompt: Answer the question using a single word or phrase. {question} . For NExT-QA, since the evaluation metric (WUPS) is sensitive to the length and the specific words used, we additionally prompt the model to be specific and respond with the most salient answer, for instance specifying “living room” instead of simply responding with “house” when asked a location question. For benchmarks that contain subtitles ( i.e. , TVQA), we include the subtitles corresponding to the clip in the prompt during inference.

We present the performance of Llama 3 8B and 70B in Table 30 . We compare Llama 3’s performance with that of two Gemini and two GPT-4 models. Note that all our results are zero-shot, as we do not include any part of these benchmarks in our training or finetuning data. We find that our Llama 3 models that train a small video adapter during post-training are very competitive, and in some cases even better, than other models that potentially leverage native multimodal processing all the way from pre-training. Llama 3 performs particularly well on video recognition given that we only evaluate the 8B and 70B parameter models. Llama 3 achieves its best performance on PerceptionTest, suggesting the model has a strong ability to perform complex temporal reasoning. On long-form activity understanding tasks like ActivityNet-QA, Llama 3 is able to obtain strong results even though it is processing only up to 64 frames, which means that for a 3-minute long video the model only processes one frame every 3 seconds.

## 8 Speech Experiments

We perform experiments to study a compositional approach of integrating speech capabilities into Llama 3, resembling the method we used for visual recognition. On the input side, an encoder, together with an adapter, is incorporated to process speech signals. We leverage a system prompt (in text) to enable different modes of operation for speech understanding in Llama 3. If no system prompt is provided, the model acts as a general-purpose spoken dialogue model which can effectively respond to the user speech in a manner that is consistent with the text-only version of Llama 3. The dialogue history is introduced as the prompt prefix to improve the multi-round dialogue experience. We also experiment with system prompts that enable the use of Llama 3 for automatic speech recognition (ASR) and automatic speech translation (AST). The speech interface of Llama 3 supports up to 34 languages. 18 18 18 The speech interface supports the following 34 languages: Arabic, Bengali, Chinese, Czech, Dutch, English, Finnish, French, German, Greek, Gujarati, Hindi, Hungarian, Indonesian, Italian, Japanese, Kannada, Korean, Malayalam, Marathi, Persian, Polish, Portuguese, Romanian, Russian, Spanish, Swahili, Swedish, Tamil, Telugu, Thai, Turkish, Urdu, Vietnamese. It also allows for the interleaved input of text and speech, enabling the model to solve advanced audio-comprehension tasks.

We also experiment with a speech generation approach in which we implement a streaming text-to-speech (TTS) system that generates speech waveforms on-the-fly during language model decoding. We design the speech generator for Llama 3 based on a proprietary TTS system and do not fine-tune the language model for speech generation. Instead, we focus on improving speech synthesis latency, accuracy, and naturalness by leveraging Llama 3 embeddings at inference time. The speech interface is illustrated in Figure 28 and 29 .

### 8.1 Data

#### 8.1.1 Speech Understanding

The training data can be categorized into two types. The pre-training data includes a large amount of unlabeled speech, which is used to initialize the speech encoder in a self-supervised manner. The supervised finetuning data includes speech recognition, speech translation, and spoken dialogue data; this data is used to unlock specific abilities when integrated with the large language model.

Pre-training data. To pre-train the speech encoder, we curate a dataset of approximately 15M hours of speech recordings encompassing a large number of languages. We filter our audio data using a voice activity detection (VAD) model and select audio samples with a VAD threshold above 0.7 for pre-training. In speech pre-training data, we also focus on ensuring the absence of PII. We use the Presidio Analyzer to identify such PII.

Speech recognition and translation data. Our ASR training data contains 230K hours of manually transcribed speech recordings that span 34 languages. Our AST training data contains 90K hours of translations in two directions: from 33 languages to English and from English to 33 languages. This data contains both supervised and synthetic data generated using the NLLB toolkit ( NLLB Team et al., 2022 ) . The use of synthetic AST data enables us to increase model quality for low-resource languages. The speech segments in our data have a maximum length of 60 seconds.

Spoken dialogue data. To finetune the speech adapter for spoken dialogue, we synthetically generate responses for speech prompts by asking the language model to respond to transcriptions of those prompts ( Fathullah et al., 2024 ) . We generate synthetic data this way using a subset of the ASR dataset with 60K hours of speech. In addition, we generate 25K hours of synthetic data by running the Voicebox TTS system ( Le et al., 2024 ) on subsets of the data used to finetune Llama 3. We used several heuristics to select a subset of finetuning data that matches the distribution of speech. These heuristics include focusing on relatively short prompts with a simple structure and without non-text symbols.

#### 8.1.2 Speech Generation

The speech generation datasets mainly consist of those for training the text normalization (TN) model and the prosody model (PM). Both training data are augmented with an additional input feature of the Llama 3 embeddings to provide contextual information.

Text normalization data. Our TN training dataset includes 55K samples that cover a wide range of semiotic classes ( e.g. , number, date, time) that require non-trivial normalization. Each sample is a pair of written-form text and the corresponding normalized spoken-form text, with an inferred sequence of handcrafted TN rules that carry out the normalization.

Prosody model data. The PM training data includes linguistic and prosodic features extracted from a 50K-hour TTS dataset, which are paired transcripts and audios recorded by professional voice actors in studio settings.