The Llama 3 Herd of Models — part 11/16

Cleaned-text code points [155787, 171019). Bibliography omitted here; source.md includes it.

We conducted extensive measurement and mitigation on a wide variety of risks to safe usage of Llama 3. However, no testing can be guaranteed to be exhaustive in identifying every possible risk. Llama 3 may still generate harmful content due to training on various datasets, particularly for languages beyond English and when prompt engineered by skilled adversarial red teamers. Malicious developers or adversarial users may find new ways to jailbreak our models and use them for various nefarious usecases. We will continue to proactively identify risks, conduct research on mitigation methods, and we encourage developers to consider responsibility in every aspect — from model development to deployment to users. We hope developers will leverage and contribute to the tools we release in our open-source system-level safety suite.

## 6 Inference

We investigate two main techniques to make inference with the Llama 3 405B model efficient: (1) pipeline parallelism and (2) FP8 quantization. We have publicly released our implementation of FP8 quantization.

### 6.1 Pipeline Parallelism

When using a BF16 number representation for the model parameters, Llama 3 405B does not fit in the GPU memory of a single machine with 8 Nvidia H100 GPUs. To address this issue, we parallelize model inference using BF16 precision across 16 GPUs on two machines. Within each machine, the high NVLink bandwidth enables the use of tensor parallelism ( Shoeybi et al., 2019 ) . Across nodes, however, connectivity has lower bandwidth and higher latency, so we use pipeline parallelism ( Huang et al., 2019 ) instead.

During training with pipeline parallelism, bubbles are a major efficiency concern (see Section 3.3 ). However, they are not an issue during inference, since inference does not involve a backward pass that requires a pipeline flush. Therefore, we use micro-batching to improve inference throughput with pipeline parallelism.

We evaluate the effect of using two micro-batches in inference workloads of 4,096 input tokens and 256 output tokens both during the key-value cache pre-fill stage of inference and during the decoding stage. We find that micro-batching improves throughput of inference with the same local batch size; see Figure 24 . These improvements result from micro-batching enabling concurrent execution of micro batches in both these stages. The additional synchronization points due to micro-batching also increase latency but, overall, micro-batching still leads to a better throughput-latency trade-off.

### 6.2 FP8 Quantization

We perform experiments leveraging the native FP8 support of H100 GPUs to perform low-precision inference. To enable low-precision inference, we apply FP8 quantization to most matrix multiplications inside the model. In particular, we quantize most parameters and activations in the feedforward network layers in the model, which account for roughly 50% of the inference compute time. We do not quantize parameters in the self-attention layers of the model. We leverage dynamic scaling factors for better accuracy ( Xiao et al., 2024b ) , optimizing our CUDA kernels 15 15 15 Our FP8 kernels are available at https://github.com/pytorch/FBGEMM/tree/main/fbgemm_gpu/experimental/gen_ai . We provide usage examples at https://github.com/meta-llama/llama-agentic-system . to reduce the overhead of calculating the scales. We find that the quality of Llama 3 405B is sensitive to certain types of quantization, and make a few additional changes to increase the model output quality:

1. Akin to Zhang et al. (2021) , we do not perform quantization in the first and last Transformer layers.

2. High-perplexity tokens such as dates can lead to large activation values. In turn, these can lead to high dynamic scaling factors in FP8 and a non-negligible number of underflows, leading to errors in decoding. To address this issue, we upper bound the dynamic scaling factors to 1200 1200 .

3. We use row-wise quantization, computing scaling factors across rows for parameter and activation matrices (see Figure 25 ). We find this works better than a tensor-wise quantization approach.

Effect of quantization errors. Evaluations on standard benchmarks often suggest that FP8 inference performs on par with BF16 inference even without these mitigations. However, we find that such benchmarks do not adequately reflect the effects of FP8 quantization. When scaling factors are not upper bounded, the model occasionally produces corrupted responses even though the benchmark performance is strong. Instead of relying on benchmarks to measure distribution changes due to quantization, we find it is better to analyze the distribution of reward-model scores for 100,000 100,000 responses produced using both FP8 and BF16. Figure 26 shows the resulting reward distribution for our quantization approach. The results in the figure show that our approach to FP8 quantization has very limited impact on the model’s response.

Experimental evaluation of efficiency. Figure 27 depicts the throughput-latency trade-off of performing FP8 inference with Llama 3 405B in the pre-fill and decoding stages, using 4,096 input tokens and 256 output tokens. The figure compares the efficiency of FP8 inference with that of the two-machine BF16 inference approach described in Section 6.1 . The results show that use of FP8 inference leads to throughput improvements of up to 50 % \% during the pre-fill stage, and a substantially better throughput-latency trade-off during decoding.

## 7 Vision Experiments

We perform a series of experiments in which we incorporate visual-recognition capabilities into Llama 3 via a compositional approach that consists of two main stages. First, we compose a pre-trained image encoder ( Xu et al., 2023 ) and the pre-trained language model by introducing and training a set of cross-attention layers between the two models ( Alayrac et al., 2022 ) on a large number of image-text pairs. This leads to the model illustrated in Figure 28 . Second, we introduce temporal aggregator layers and additional video cross-attention layers that operate on a large collection of video-text pairs to learn the model to recognize and process temporal information from videos.

A compositional approach to foundation model development has several advantages: (1) it enables us to parallelize the development of the vision and language modeling capabilities; (2) it circumvents complexities of joint pre-training on visual and language data that stem from tokenization of visual data, differences in background perplexities of tokens originating from different modalities, and contention between modalities; (3) it guarantees that model performance on text-only tasks is not affected by the introduction of visual-recognition capabilities, and (4) the cross-attention architecture ensures that we do not have to expend compute passing full-resolution images through the increasingly LLM backbones (specifically, the feed-forward networks in each transformer layer), making it more efficient during inference. We note that our multimodal models are still under development and not yet ready for release.

Before presenting the results of our experiments in Section 7.6 and 7.7 , we describe the data we used to train visual recognition capabilities, the model architecture of the vision components, how we scale training of those components, and our pre-training and post-training recipes.

### 7.1 Data

We describe our image and video data separately below.

#### 7.1.1 Image Data

Our image encoder and adapter are trained on image-text pairs. We construct this dataset via a complex data processing pipeline that consists of four main stages: (1) quality filtering, (2) perceptual de-duplication, (3) resampling, and (4) optical character recognition. We also apply a series of safety mitigations.

• Quality filtering. We implement quality filters that remove non-English captions and low-quality captions via heuristics such as low alignment scores produced by ( Radford et al., 2021 ) . Specifically, we remove all image-text pairs below a certain CLIP score.

• De-duplication. De-duplicating large-scale training datasets benefits model performance because it reduces training compute spent on redundant data ( Esser et al., 2024 ; Lee et al., 2021 ; Abbas et al., 2023 ) and memorization ( Carlini et al., 2023 ; Somepalli et al., 2023 ) . Hence, we de-duplicate our training data for both efficiency and privacy reasons. To do so, we use an internal version of the state-of-the-art SSCD copy-detection model ( Pizzi et al., 2022 ) to de-duplicate images at scale. For all images, we first compute a 512-dimensional representation using the SSCD model. We use those embeddings to perform a nearest neighbor (NN) search for each image across all images in our data set, using a cosine similarity measure. We define examples above a certain similarity threshold as duplicates. We group these duplicates using a connected-components algorithm, and maintain only one image-text pair per connected component. We increase the efficiency of our de-duplication pipeline by: (1) pre-clustering the data using k-means clusters and (2) using FAISS ( Johnson et al., 2019 ) for NN searches and clustering.

• Resampling. We ensure diversity of the image-text pairs via resampling akin to Xu et al. (2023) ; Mahajan et al. (2018) ; Mikolov et al. (2013) . First, we construct a vocabulary of n-grams by parsing high-quality text sources. Next, we compute the frequency of each vocabulary n-gram in our dataset. We then resample the data as follows: If any of the n-grams in a caption occurs less than T T times in the vocabulary, we keep the corresponding image-text pair. Otherwise, we independently sample each of the n-grams n i n_{i} in the caption with probability T / f i \sqrt{T/f_{i}} where f i f_{i} indicates the frequency of n-gram n i n_{i} ; we keep the image-text pair if any of the n-grams was sampled. This resampling aids performance on low-frequency categories and fine-grained recognition tasks.

• Optical character recognition. We further improve our image-text data by extracting text written in the image and concatenating it with the caption. The written text is extracted using a proprietary optical character recognition (OCR) pipeline. We observe that adding OCR data into the training data greatly improves tasks that require OCR capabilities, such as document understanding.

Transcribing documents. To improve the performance of our models on document understanding tasks, we render pages from documents as images and paired the images with their respective text. The document text is obtained either directly from the source or via a document parsing pipeline.

Safety. We focus primarily on ensuring that the pre-training dataset for image recognition does not contain unsafe content, such as sexual abuse material (CSAM) ( Thiel, 2023 ) . We scan all our training images for CSAM using perceptual hashing approaches such as PhotoDNA ( Farid, 2021 ) as well as internal, proprietary classifiers. We also use a proprietary media-risk retrieval pipeline to identify and remove image-text pairs that we consider to be NSFW, for example, because they contain sexual or violent content. We believe that minimizing the prevalence of such material in the training dataset improves the safety of the final model without impacting its helpfulness. Finally, we perform face blurring on all images in our training set. We test the model against human generated prompts that refer to an attached image.

Annealing data. We create an annealing dataset by resampling the image-caption pairs to a smaller volume of ∼ \sim 350M examples using n-grams. Since the n-grams resampling favor richer text descriptions, this selects a higher-quality data subset. We augment the resulting data with ∼ \sim 150M examples from five additional sources:

• Visual grounding. We link noun phrases in the text to bounding boxes or masks in the image. The grounding information (bounding boxes and masks) are specified in the image-text pair in two ways. (1) We overlay boxes or masks with marks on the image and use marks in the text as reference, akin to set-of-marks ( Yang et al., 2023a ) . (2) We insert normalized ( x min , y min , x max , y max ) (x_{\textrm{min}},y_{\textrm{min}},x_{\textrm{max}},y_{\textrm{max}}) coordinates directly into the text, demarcated by special tokens.

• Screenshot parsing. We render screenshots from HTML code and task the model with predicting the code that produced a specific element in the screenshot, akin to Lee et al. (2023) . The element of interest is indicated in the screenshot via a bounding box.

• Question-answer pairs. We include question-answer pairs, enabling us to use volumes of question-answering data that are too large to be used in model finetuning.

• Synthetic captions. We include images with synthetic captions that were generated by an early version of the model. Compared to original captions, we find that synthetic captions provide a more comprehensive description of images than the original captions.

• Synthetically-generated structured images. We also include synthetically generated images for a variety of domains such as charts, tables, flowcharts, math equations and textual data. These images are accompanied by a structured representation such as the corresponding markdown or LaTeX notation. Besides improving recognition capabilities of the model for these domains, we find this data useful to generate question-answer pairs via the text model for finetuning.

#### 7.1.2 Video Data

For video pre-training, we use a large dataset of video-text pairs. Our dataset is curated through a multi-stage process. We filter and clean the associated texts using rule-based heuristics, such as ensuring a minimum length and fixing capitalization. Then, we run language identification models to filter out non-English texts. We run OCR detection models to filter out videos with excessive overlaid text. To ensure reasonable alignment between the video-text pairs, we use CLIP ( Radford et al., 2021 ) style image-text and video-text contrastive models. We first compute image-text similarity using a single frame in the videos and filtered out low similarity pairs, and then subsequently filter out pairs with low video-text alignment. Some of our data contains static or low-motion videos; we filter out such data using motion-score based filtering ( Girdhar et al., 2023 ) . We do not apply any filters on the visual quality of the videos such as aesthetic scores or resolution filtering.

Our dataset contains videos with an average duration of 21 seconds and a median duration of 16 seconds, with over 99 % 99\% videos being under a minute. The spatial resolution varies significantly between 320p and 4K videos, with over 70 % 70\% of the videos having a short side greater than 720 pixels. The videos have varying aspect ratios with almost all videos having between aspect ratio between 1 : 2 1{:}2 and 2 : 1 2{:}1 , with a 1 : 1 1{:}1 median.

### 7.2 Model Architecture

Our visual-recognition model consists of three main components: (1) an image encoder, (2) an image adapter, and (3) a video adapter.