The Llama 3 Herd of Models — part 4/16

Cleaned-text code points [46535, 62494). Bibliography omitted here; source.md includes it.

In the final stages of pre-training, we train on long sequences to support context windows of up to 128K tokens. We do not train on long sequences earlier because the compute in self-attention layers grows quadratically in the sequence length. We increase the supported context length in increments, pre-training until the model has successfully adapted to the increased context length. We assess successful adaptation by measuring whether (1) model performance on short-context evaluations has recovered completely and (2) the model perfectly solves “needle in a haystack” tasks up to that length. In Llama 3 405B pre-training, we increased context length gradually in six stages, starting from the original 8K context window and ending in the final 128K context window. This long-context pre-training stage was performed using approximately 800B training tokens.

#### 3.4.3 Annealing

During pre-training on the final 40M tokens, we linearly annealed the learning rate to 0, maintaining a context length of 128K tokens. During this annealing phase, we also adjusted the data mix to upsample data sources of very high quality; see Section 3.1.3 . Finally, we compute the average of model checkpoints ( Polyak (1991) averaging) during annealing to produce the final pre-trained model.

## 4 Post-Training

We produce the aligned Llama 3 models by applying several rounds of post-training, 6 6 6 We use the term “post-training” to refer to any model training that happens outside of pre-training. or aligning the model with human feedback ( Ouyang et al., 2022 ; Rafailov et al., 2024 ) on top of a pre-trained checkpoint. Each round of post-training involves supervised finetuning (SFT) followed by Direct Preference Optimization ( Rafailov et al., 2024 , DPO;) on examples collected either via human annotations or generated synthetically. Our post-training modeling and data approaches are described in Sections 4.1 and 4.2 respectively. We further detail custom data curation strategies to improve the reasoning, coding, factuality, multilingual, tool use, long context, and precise instruction following in Section 4.3 .

### 4.1 Modeling

The backbone of our post-training strategy is a reward model and a language model. We first train a reward model on top of the pre-trained checkpoint using human-annotated preference data (see Section 4.1.2 ). We then finetune pre-trained checkpoints with supervised finetuning (SFT; see Section 4.1.3 ), and further align the checkpoints with Direct Preference Optimization (DPO; see Section 4.1.4 ). This process is illustrated in Figure 7 . Unless otherwise noted, our modeling procedure applies to Llama 3 405B, and we refer to Llama 3 405B as Llama 3 for simplicity.

#### 4.1.1 Chat Dialog Format

To tune LLMs for human-AI interaction, we need to define a chat dialog protocol for the model to understand human instructions and perform conversational tasks. Compared to its predecessor, Llama 3 has new capabilities such as tool use (Section 4.3.5 ) which may require generating multiple messages and sending them to different locations (e.g., user, ipython ) within a single dialog turn. To support this, we design a new multi-message chat protocol which uses various special header and termination tokens. The header tokens are used to indicate the source and destination of each message in a conversation. Similarly, the termination tokens indicate when it is the time to alternate between human and AI to speak.

#### 4.1.2 Reward Modeling

We train a reward model (RM) covering different capabilities on top of the pre-trained checkpoint. The training objective is the same as Llama 2 except that we remove the margin term in the loss, as we observe diminishing improvements after data scaling. Following Llama 2, we use all of our preference data for reward modeling after filtering out samples with similar responses. In addition to standard preference pair of (chosen, rejected) response, annotations also create a third “edited response” for some prompts, where the chosen response from the pair is further edited for improvement (see Section 4.2.1 ). Hence, each preference ranking sample has two or three responses with clear ranking ( edited ¿ chosen ¿ rejected ). We concatenate the prompt and multiple responses into a single row during training with responses randomly shuffled. This is an approximation to the standard scenario of putting the responses in separate rows and computing the scores, but in our ablations, this approach improves training efficiency without a loss in accuracy.

#### 4.1.3 Supervised Finetuning

The reward model is then used to perform rejection sampling on our human annotation prompts, the details of which are described in Section 4.2 . Together with this rejection-sampled data and other data sources (including synthetic data), we finetune the pre-trained language model using a standard cross entropy loss on the target tokens (while masking loss on prompt tokens). More details about the data mix can be found in Section 4.2 . We refer to this stage as supervised finetuning ( Wei et al., 2022a ; Sanh et al., 2022 ; Wang et al., 2022b , SFT;) , even though many of the training targets are model-generated. Our largest models are finetuned with a learning rate of 10 − 5 10^{-5} over the course of 8.5K to 9K steps. We found these hyperparameter settings to work well across different rounds and data mixes.

#### 4.1.4 Direct Preference Optimization

We further train our SFT models with Direct Preference Optimization ( Rafailov et al., 2024 , DPO;) for human preference alignment. For training, we primarily use the most recent batches of preference data collected using the best performing models from the previous alignment rounds. As a result, our training data conforms better to the distribution of the policy model that is being optimized in each round. We also explored on-policy algorithms such as PPO ( Schulman et al., 2017 ) , but found that DPO required less compute for large-scale models and performed better, especially on instruction following benchmarks like IFEval ( Zhou et al., 2023 ) . For Llama 3, we use a learning rate of 10 − 5 10^{-5} and set the β \beta hyper-parameter to be 0.1. In addition, we apply the following algorithmic modifications to DPO:

• Masking out formatting tokens in DPO loss : We mask out special formatting tokens including header and termination tokens (described in Section 4.1.1 ) from both chosen and rejected responses in the loss to stabilize DPO training. We observe that having these tokens contribute to the loss may lead to undesired model behaviors such as tail repetition or abruptly generating termination tokens. We hypothesize that this is due to the contrastive nature of the DPO loss – the presence of common tokens in both chosen and rejected responses leads to a conflicting learning objective as the model needs to increase and reduce the likelihood of these tokens simultaneously.

• Regularization with NLL loss : We add an additional negative log-likelihood (NLL) loss term with a scaling coefficient of 0.2 0.2 on the chosen sequences, similar to Pang et al. (2024) . This helps further stabilize DPO training by maintaining desired formatting for generation and preventing the decrease of log probability of chosen responses ( Pang et al., 2024 ; Pal et al., 2024 ) .

#### 4.1.5 Model Averaging

Finally, we average models obtained from experiments using various versions of data or hyperparameters at each RM, SFT, or DPO stage ( Izmailov et al., 2019 ; Wortsman et al., 2022 ; Li et al., 2022 ) .

#### 4.1.6 Iterative Rounds

Following Llama 2, we apply the above methods in six rounds. In each cycle, we collect new preference annotations and SFT data, sampling synthetic data from the latest models.

### 4.2 Post-training Data

The post-training data composition plays a critical role in the usefulness and behavior of language models. In this section, we discuss our human annotation procedures and preference data collection (Section 4.2.1 ), the composition of our SFT data (Section 4.2.2 ), and methods for data quality control and cleaning (Section 4.2.3 ).

#### 4.2.1 Preference Data

Our preference data annotation process is similar to Llama 2. We deploy multiple models for annotation after each round and sample two responses from two different models for each user prompt. These models can be trained with different data mixes and alignment recipes, allowing for different capability strength ( e.g. , code expertise) and increased data diversity. We ask annotators to rate the strength of their preference by categorizing it into one of four levels, based on how much more they prefer the chosen response over the rejected one: significantly better, better, slightly better, or marginally better. We also incorporate an editing step after preference ranking to encourage annotators to further improve the preferred response. Annotators edit the chosen response directly or prompt the model with feedback to refine its own response. Consequently, a portion of our preference data has three responses ranked ( edited ¿ chosen ¿ rejected ).

In Table 6 , we report the statistics of preference annotations that we use for Llama 3 training. General English covers multiple subcategories such as knowledge-based question and answering or precise instruction-following, which fall outside the scope of specific capabilities. Compared to Llama 2, we observe an increase in the average length of prompt and response, suggesting that we train Llama 3 on more complex tasks. In addition, we implement a quality analysis and human evaluation process to rigorously assess the data collected, allowing us to refine our prompts and provide systematic, actionable feedback to annotators. For example, as Llama 3 improves after each round, we increase prompt complexity accordingly to target areas where the model lags.

In each round of post-training, we use all the preference data that is available at the time for reward modeling, while only using the latest batches from various capabilities for DPO training. For both reward modeling and DPO, we use samples that are labeled as the chosen response being significantly better or better than the rejected counterpart for training and discard samples with similar responses.

#### 4.2.2 SFT Data

Our finetuning data is largely comprised of the following sources:

• Prompts from our human annotation collection with rejection-sampled responses.

• Synthetic data targeting specific capabilities (see Section 4.3 for more details).

• Small amounts of human-curated data (see Section 4.3 for more details).

As our post-training rounds progress, we develop stronger Llama 3 variants that we use to collect larger datasets that cover a wide range of complex capabilities. In this section, we discuss the details for the rejection-sampling procedure and overall composition of our final SFT datamix.

Rejection sampling. During rejection sampling (RS), for each prompt collected during human annotation (Section 4.2.1 ) we sample K K (typically between 10 and 30) outputs from the latest chat model policy (usually the best performing checkpoint from the previous post-training iteration, or the best performing checkpoint for a particular capability) and use our reward model to select the best candidate, consistent with Bai et al. (2022) . In later rounds of post-training, we introduce system prompts to steer RS responses to conform with desirable tone, style, or formatting, which might be different for different capabilities.

To increase the efficiency of rejection sampling, we adopt PagedAttention ( Kwon et al., 2023 ) . PagedAttention enhances memory efficiency through dynamic key-value cache allocation. It supports arbitrary output lengths by dynamically scheduling requests based on the current cache capacity. Unfortunately, this carries the risk of swap-out when running out of memory. To eliminate such swap overhead, we define a maximum output length and perform a request only if sufficient memory is available to fit an output with that length. PagedAttention also enables us to share the key-value cache pages for a prompt across all corresponding outputs. Together, this leads to a throughput improvement of over 2 × 2\times during rejection sampling.

Overall data composition. Table 7 shows data statistics for each broad category of our “helpfulness” mix. While SFT and preference data contain overlapping domains, they are curated differently, yielding distinct count statistics. In Section 4.2.3 we describe techniques for categorizing topic, complexity, and quality of our data samples. In each round of post-training, we adjust our overall data mix carefully across these axes to tune performance across a wide range of benchmarks. Our final data mix epochs multiple times on some high quality sources and downsamples others.

#### 4.2.3 Data Processing and Quality Control

Given that most of our training data is model-generated , it requires careful cleaning and quality control.

Data cleaning. In the early rounds, we observed a number of undesirable patterns common in our data, such as excessive use of emojis or exclamation points. Therefore, we implement a series of rule-based data removal and modification strategies to filter or clean problematic data. For example, to mitigate overly-apologetic tonal issues, we identify overused phrases (such as “I’m sorry” or “I apologize”) and carefully balance the proportion of such samples in our dataset.

Data pruning. We also apply a collection of model-based techniques to remove low-quality training samples and improve overall model performance:

• Topic classification: We first finetune Llama 3 8B into a topic classifier, and perform inference over all data to classify it into both coarsely-grained buckets (“mathematical reasoning”) and fine-grained buckets (“geometry and trigonometry”).

• Quality scoring: We use both reward model and Llama-based signals to obtain a quality score for each sample. For an RM-based score, we consider data that is in the top quartile of RM scores as high quality. For a Llama-based score, we prompt Llama 3 checkpoint to rate each sample on a three-point scale for general English data (accuracy, instruction following, and tone/presentation) and a two-point scale for coding data (bug identification and user intention), and consider samples that obtain the maximum score as high quality. The RM and Llama-based scores have high disagreement rates, and we find that combining these signals yield the best recall on our internal test set. Ultimately, we select examples that are marked as high quality by the RM or the Llama-based filter.

• Difficulty scoring: Because we are also interested in prioritizing examples that are more complex for the model, we score data using two measures of difficulty: Instag ( Lu et al., 2023 ) and Llama-based scoring. For Instag, we prompt Llama 3 70B to perform intention tagging of SFT prompts, where more intentions implies more complexity. We also prompt Llama 3 to measure the difficulty ( Liu et al., 2024c ) of dialogs on a three-point scale.

• Semantic deduplication: Finally, we perform semantic deduplication ( Abbas et al., 2023 ; Liu et al., 2024c ) . We first cluster complete dialogs using RoBERTa ( Liu et al., 2019b ) and within each cluster sort them by quality score × \times difficulty score. We then do greedy selection by iterating through all sorted examples, and only keeping the ones that have maximum cosine similarity less than a threshold to the examples seen so far in the cluster.

### 4.3 Capabilities

We highlight special efforts to improve performance for specific capabilities such as code (Section 4.3.1 ), multilinguality (Section 4.3.2 ), math and reasoning (Section 4.3.3 ), long context (Section 4.3.4 ), tool use (Section 4.3.5 ), factuality (Section 4.3.6 ), and steerability (Section 4.3.7 ).

#### 4.3.1 Code