The Llama 3 Herd of Models — part 12/16

Cleaned-text code points [171019, 186342). Bibliography omitted here; source.md includes it.

Image encoder. Our image encoder is a standard vision transformer (ViT; Dosovitskiy et al. (2020) ) that is trained to align images and text ( Xu et al., 2023 ) . We use the ViT-H/14 variant of the image encoder, which has 630M parameters that were trained on 2.5B image-text pairs for five epochs. The image encoder is pre-trained on images with resolution 224 × 224 224\times 224 ; images were split up into 16 × 16 16\times 16 patches of equal size ( i.e. , a patch size of 14 ​ x ​ 14 14x14 pixels). As also demonstrated by prior work such as ViP-Llava ( Cai et al., 2024 ) , we observe that image encoders trained via a contrastive text alignment objective are unable to preserve fine-grained localization information. To alleviate this, we employ a multi-layer feature extraction, where features from the 4 th , 8 th , 16 th , 24 th and 31 st layers are also provided in addition to the final layer features. In addition, we further insert 8 gated self-attention layers (making a total of 40 transformer blocks) prior to pre-training of the cross-attention layers to learn alignment-specific features. The image encoder therefore eventually has a total 850 850 M parameters with the additional layers. With the multi-layer features, the image encoder produces a 7680 7680 -dimensional representation for each of the resulting 16 × = 256 16\times 16\!=\!256 patches. The parameters of the image encoder are not frozen during subsequent training stages as we found it to improve performance, especially in domains such as text recognition.

Image adapter. We introduce cross-attention layers between the visual token representations produced by the image encoder and the token representations produced by the language model ( Alayrac et al., 2022 ) . The cross-attention layers are applied after every fourth self-attention layer in the core language model. Like the language model itself, the cross-attention layers use generalized query attention (GQA) for increased efficiency. The cross-attention layers introduce substantial numbers of additional trainable parameters into the model: for Llama 3 405B, the cross-attention layers have ≈ \approx 100B parameters. We pre-train our image adapter in two stages: (1) initial pre-training followed by (2) annealing: • Initial pre-training. We pre-train our image adapter on our dataset of ∼ \sim 6B image-text pairs described above. For compute efficiency reasons, we resize all images to fit within at most four tiles of 336 × 336 336\times 336 pixels each, where we arrange the tiles to support different aspect ratios, e.g. , 672 × 672 672\times 672 , 672 × 336 672\times 336 , and 1344 × 336 1344\times 336 .

• Annealing. We continue training the image adapter on ∼ \sim 500M images from the annealing dataset described above. During annealing, we increase the per-tile image resolution to improve performance on tasks that require higher-resolution images, for example, infographics understanding.

Video adapter. Our model takes as input up to 64 frames (uniformly sampled from a full video), each of which is processed by the image encoder. We model temporal structure in videos through two components: (i) encoded video frames are aggregated by a temporal aggregator which merges 32 consecutive frames into one, (ii) additional video cross attention layers are added before every fourth image cross attention layer. The temporal aggregator is implemented as a perceiver resampler ( Jaegle et al., 2021 ; Alayrac et al., 2022 ) . We pre-train using 16 frames per video (aggregated to 1 frame), but increase the number of input frames to 64 during supervised finetuning. The video aggregator and cross attention layers have 0.6B and 4.6B parameters for Llama 3 7B and 70B, respectively.

### 7.3 Model Scaling

After the visual-recognition components are added to Llama 3, the model contains self-attention layers, cross-attention layers, and a ViT image encoder. To train adapters for the smaller 8B and 70B parameter models, we found a combination of data and tensor parallelization is the most efficient. Model or pipeline parallelism does not increase efficiency at these scales because the gathering of model parameters would dominate the computation. We do, however, use pipeline parallelism (in addition to data and tensor parallelism) when training the adapter for the 405B parameter model. Training at this scale introduces three new challenges in addition to those outlined in Section 3.3 : model heterogeneity, data heterogeneity, and numerical instabilities.

Model heterogeneity. The model computation is heterogeneous because more computation is performed on some tokens than on others. In particular, image tokens are processed by the image encoder and the cross-attention layers, whereas text tokens are only processed by the language backbone. This heterogeneity leads to bottlenecks in the scheduling of pipeline parallelism. We address this problem by ensuring each pipeline stage contains five layers: namely, four self-attention layers in the language backbone and a cross-attention layer. (Recall that we introduce a cross-attention layer after every fourth self-attention layer.) In addition, we replicate the image encoder on all pipeline stages. Because we train on paired image-text data, this enables us to perform load balancing between the image and text parts of the computation.

Data heterogeneity. The data is heterogeneous because, on average, images have more tokens than the associated text: an image has 2,308 tokens, whereas the associated text contains an average of only 192 tokens. As a result, the computation of cross-attention layers requires more time and memory than the computation of self-attention layers. We address this problem by introducing sequence parallelization in the image encoder, so that each GPU processes roughly the same number of tokens. Because the average text size is relatively short, we also use a substantially larger micro-batch size (8 instead of 1).

Numerical instabilities. After the image encoder is added to the model, we find that performing gradient accumulation in bf16 led to numerical instabilities. The most likely explanation for this is that image tokens are introduced into the language backbone via all cross-attention layers. This implies that numerical deviations in the representation of an image token have an outsized impact on the overall computation because the errors are compounded. We address this by performing gradient accumulation in FP32.

### 7.4 Pre-training

Image. We initialize from the pre-trained text model and vision encoder weights. The vision encoder is unfrozen, while the text model weights are kept frozen as explained above. First, we train the model using 6B image-text pairs where each image is resized to fit within four tiles of 336 × 336 336\times 336 pixels. We use a global batch size of 16,384 and a cosine learning rate schedule with initial learning rate 10 × 10 − 4 10\times 10^{-4} and a weight decay of 0.01 0.01 . The initial learning rate was determined based on small-scale experiments. However, these findings did not generalize well to very long training schedules and dropped the learning rate a few times during training when the loss values became stagnant. After the base pre-training, we increase the image resolution further and continue training the same weights on the annealing dataset. The optimizer is re-initialized via warm-up to learning rate 2 × 10 − 5 2\times 10^{-5} and again follows a cosine schedule.

Video. For video pre-training, we start from the image pre-trained and annealed weights as described above. We add the video aggregator and cross-attention layers as described in the architecture, initialized randomly. We freeze all the parameters in the model except the video-specific ones (the aggregator and video cross-attention), and train them on the video pre-training data. We use the same training hyperparameters as the image annealing stage, with small differences in the learning rate. We uniformly sample 16 frames from the full video, and represent each frame using four chunks, each of size of 448 × 448 448\times 448 pixels. We use an aggregation factor of 16 in the video aggregator, hence obtaining one effective frame, which the text tokens cross-attend to. We use a global batch size of 4,096, a sequence length of 190 tokens, and a learning rate of 10 − 4 10^{-4} during training.

### 7.5 Post-Training

In this section, we describe the post-training recipe for our vision adapters. After pre-training, we fine-tune the model on highly curated multi-modal conversational data to enable chat capabilities. We further implement direct preference optimization (DPO) to boost human evaluation performance and rejection sampling to improve multi-modal reasoning capabilities. Finally, we add a quality-tuning stage where we continue fine-tuning the model on a very small set of high-quality conversational data which further boosts human evaluation while retaining performance across benchmarks. More details on each of these steps are provided below.

#### 7.5.1 Supervised Finetuning Data

We describe our supervised finetuning (SFT) data for image and video capabilities separately below.

Image. We utilize a mix of different datasets for supervised finetuning. • Academic datasets. We convert a highly filtered collection of existing academic datasets to question-answer pairs using templates or via LLM rewriting. The LLM rewriting’s purpose is to augment the data with different instructions and to improve the language quality of answers.

• Human annotations. We collect multi-modal conversation data via human annotators for a wide range of tasks (open-ended question-answering, captioning, practical use cases, etc. ) and domains ( e.g. , natural images and structured images). Annotators are provided with images and asked to write conversations. To ensure diversity, we cluster large-scale datasets and sampled images uniformly across different clusters. Further, we acquire additional images for a few specific domains by expanding a seed via k-nearest neighbors. Annotators are also provided with intermediate checkpoints of existing models to facilitate model-in-the-loop style annotations, so that model generations can be utilized as a starting point by the annotators to then provide additional human edits. This is an iterative process, in which model checkpoints would be regularly updated with better performing versions trained on the latest data. This increases the volume and efficiency of human annotations, while also improving their quality.

• Synthetic data. We explore different ways to generate synthetic multi-modal data by using text-representations of images and a text-input LLM. The high-level idea is to utilize the reasoning capabilities of text-input LLMs to generate question-answer pairs in the text domain, and replace the text representation with its corresponding images to produce synthetic multi-modal data. Examples include rendering texts from question-answer datasets as images or rendering table data into synthetic images of tables and charts. Additionally, we use captions and OCR extractions from existing images to generate additional conversational or question-answer data related to the images.

Video. Similar to the image adapter, we use academic datasets with pre-existing annotations and convert them into appropriate textual instructions and target responses. The targets are converted to open-ended responses or multiple-choice options, whichever is more appropriate. We ask humans to annotate videos with questions and corresponding answers. The annotators are asked to focus on questions that could not be answered based on a single frame, to steer the annotators towards questions that require temporal understanding.

#### 7.5.2 Supervised Finetuning Recipe

We describe our supervised finetuning (SFT) recipe for image and video capabilities separately below.

Image. We initialize from the pre-trained image adapter, but hot-swap the pre-trained language model’s weights with the instruction tuned language model’s weights. The language model weights are kept frozen to maintain text-only performance, i.e. , we only update the vision encoder and image adapter weights.

Our approach to finetune the model is similar to Wortsman et al. (2022) . First, we run a hyperparameter sweep using multiple random subsets of data, learning rates and weight decay values. Next, we rank the models based on their performance. Finally, we average the weights of the top- K K models to obtain the final model. The value of K K is determined by evaluating the averaged models and selecting the instance with highest performance. We observe that the averaged models consistently yield better results compared to the best individual model found via grid search. Further, this strategy reduces sensitivity to hyperparameters.

Video. For video SFT, we initialize the video aggregator and cross-attention layers using the pre-trained weights. The rest of the parameters in the model, the image weights and the LLM, are initialized from corresponding models following their finetuning stages. Similar to video pre-training, we then finetune only the video parameters on the video SFT data. For this stage, we increase the video length to 64 frames, and use an aggregation factor of 32 to get two effective frames. The resolution of the chunks is also increased to be consistent with the corresponding image hyperparameters.

#### 7.5.3 Preference Data

We built multimodal pair-wise preference datasets for reward modeling and direct preference optimization. • Human annotations. The human-annotated preference data consists of comparisons between two different model outputs, labeled as “chosen” and “rejected”, with 7-scale ratings. The models used to generate responses are sampled on-the-fly from a pool of the best recent models, each with different characteristics. We update the model pool weekly. Besides preference labels, we also request annotators to provide optional human edits to correct inaccuracies in “chosen” responses because vision tasks have a low tolerance for inaccuracies. Note that human editing is an optional step because there is a trade-off between volume and quality in practice.

• Synthetic data. Synthetic preference pairs could also be generated by using text-only LLMs to edit and deliberately introduce errors in the supervised finetuning dataset. We took the conversational data as input, and use an LLM to introduce subtle but meaningful errors ( e.g. , change objects, change attributes, add mistakes in calculations, etc.). These edited responses are used as negative “rejected” samples and paired with the “chosen” original supervised finetuning data.

• Rejection sampling. Furthermore, to create more on-policy negative samples, we leveraged the iterative process of rejection sampling to collect additional preference data. We discuss our usage of rejection sampling in more detail in the following sections. At a high-level, rejection sampling is used to iteratively sample high-quality generations from a model. Therefore, as a by-product, all generations that are not selected can be used as negative rejected samples and used as additional preference data pairs.

#### 7.5.4 Reward Modeling