The Llama 3 Herd of Models — part 7/16

Cleaned-text code points [94313, 110218). Bibliography omitted here; source.md includes it.

Modeling. After we collect the preference data, we leverage this data in reward modeling, rejection sampling, SFT, and DPO to enhance Llama 3’s steerability.

## 5 Results

We performed an extensive series of evaluations of Llama 3, investigating the performance of: (1) the pre-trained language model, (2) the post-trained language model, and (3) the safety characteristics of Llama 3. We present the results of these evaluations in separate subsections below.

### 5.1 Pre-trained Language Model

In this section, we report evaluation results for our pre-trained Llama 3 (Section 3 ), comparing with various other models of comparable sizes. We reproduce results of competitor models whenever possible. For non-Llama models, we report the best score across results that are publicly reported or (where possible) that we reproduced ourselves. The specifics of these evaluations, including configurations such as the number of shots, metrics, and other pertinent hyperparameters and settings, can be accessed on our Github repository here. Additionally, we are releasing the data generated as part of evaluations with publicly available benchmarks which can be found on Huggingface here . We evaluate the quality of our models on standard benchmarks (Section 5.1.1 ), for robustness to changes in multiple-choice question setups (Section 5.1.2 ), and on adversarial evaluations (Section 5.1.3 ). We also conduct a contamination analysis to estimate the extent to which our evaluations are impacted by contamination of training data (Section 5.1.4 ).

#### 5.1.1 Standard Benchmarks

To compare our models with the current state-of-the-art, we evaluate Llama 3 on a large number of standard benchmark evaluations shown in Table 8 . These evaluations cover eight top-level categories: (1) commonsense reasoning; (2) knowledge; (3) reading comprehension; (4) math, reasoning, and problem solving; (5) long context; (6) code; (7) adversarial evaluations; and (8) aggregate evaluations.

Experimental setup. For each benchmark, we compute scores for Llama 3 as well as various other pre-trained models of comparable sizes. Where possible, we recompute numbers with our own pipeline for other models. To ensure a fair comparison, we then select the best score between the score that we computed and the reported number for that model with comparable or more conservative settings. You can find additional details on our evaluation setup here . For some models, it is not possible to (re)compute benchmark values, for instance, because the pre-trained model is not released or because the API does not provide access to log-probabilities. In particular, this is true for all models comparable to Llama 3 405B. Thus, we do not report category averages for Llama 3 405B, which requires that all numbers are available for all benchmarks.

Significance estimates. Benchmark scores are estimates of a model’s true performance. These estimates have variance because benchmark sets are finite samples drawn from some underlying distribution. We follow Madaan et al. (2024b) and report on this variance via 95% confidence intervals (CIs), assuming that benchmark scores are Gaussian distributed. While this assumption is incorrect ( e.g. , benchmark scores are bounded), preliminary bootstrap experiments suggest CIs (for discrete metrics) are a good approximation: C ​ I ​ ( S ) = 1.96 × S × ( 1 − S ) N . CI(S)=1.96\times\sqrt{\frac{S\times(1-S)}{N}}. Herein, S S is the observed benchmark score ( e.g. , accuracy or EM) and N N the sample size of the benchmark. We omit CIs for benchmark scores that are not simple averages. We note that because subsampling is not the only source of variation, our CI values lower bound the actual variation in the capability estimate.

Results for 8B and 70B models. Figure 12 reports the average performance of Llama 3 8B and 70B on the commonsense reasoning, knowledge, reading comprehension, math and reasoning, and code benchmarks. The results show that Llama 3 8B outperforms competing models in virtually every category, both in terms of per-category win rate and in terms of average per-category performance. We also find that Llama 3 70B outperforms its predecessor Llama 2 70B by a large margin on most benchmarks, with the exception of commonsense benchmarks that are likely saturated. Llama 3 70B also outperforms Mixtral 8x22B.

Detailed results for all models. Table 10 , 10 , 11 , 12 , 13 , and 15 present the benchmark performance of pre-trained Llama 3 8B, 70B, and 405B models on reading comprehension tasks, coding tasks, commonsense understanding tasks, mathematical reasoning tasks, and general tasks. The tables compare Llama 3’s performance with that of models of similar size. The results show that Llama 3 405B performs competitively with other models in its class. In particular, Llama 3 405B substantially outperforms prior open-source models. For long-context, we present more comprehensive results (including probing tasks like needle-in-a-haystack) in Section 5.2 .

#### 5.1.2 Model Robustness

In addition to performance on benchmarks, robustness is an important factor in the quality of pre-trained language models. We investigate the robustness of our pre-trained language models to design choices in multiple-choice question (MCQ) setups. Prior work has reported that model performance can be sensitive to seemingly arbitrary design choices in such setups, for example, model scores and even rankings may change with the order and labels of the in-context examples ( Lu et al., 2022 ; Zhao et al., 2021 ; Robinson and Wingate, 2023 ; Liang et al., 2022 ; Gupta et al., 2024 ) , the exact format of the prompt ( Weber et al., 2023b ; Mishra et al., 2022 ) , or the answer choice format and order ( Alzahrani et al., 2024 ; Wang et al., 2024a ; Zheng et al., 2023 ) . Motivated by this work, we use the MMLU benchmark to evaluate the robustness of our pre-trained models to: (1) few-shot label bias, (2) label variants, (3) answer order, and (4) prompt format:

• Few-shot label bias. Following Zheng et al. (2023) and Weber et al. (2023a) , we investigate the impact of the distribution of labels in four-shot examples. Specifically, we consider settings in which: (1) all few-shot examples have the same label ( A A A A ); (2) all examples have a different label ( A B C D ); and (3) there are only two labels present ( A A B B and C C D D ).

• Label variants. We also study model response to different choice token sets. We consider the two sets proposed by Alzahrani et al. (2024) : namely, a set of common language independent tokens ( $ & # @ ) and a of rare tokens ( œ § russianз ü ) that do not have any implicit relative order. We also consider two versions of the canonical labels ( A. B. C. D. and A) B) C) D) ) and a numerical list ( 1. 2. 3. 4. ).

• Answer order. Following Wang et al. (2024a) , we compute how stable the results are across different answer orders. To compute this, we remap all the answers in the dataset according to a fixed permutation. For example, for the permutation A B C D , all answer options with label A and B keep their label, and all answer options with label C get label D , and vice versa.

• Prompt format. We evaluate variance in performance across five task prompts that differ in the level of information provided: one prompt simply asks the model to answer the question, whereas other prompts assert the expertise of the model or that the best answer should be chosen.

Figure 13 presents the results of our experiments studying robustness of model performance to label variants (left) and few-shot label bias (right). The results show that our pre-trained language models are very robust to changes in MCQ labels and to the structure of the few-shot prompt labels. This robustness is particularly pronounced for the 405B parameter model. Figure 14 presents the results of our study of robustness to answer order and prompt format. The results in the figure further underscore the robustness of the performance of our pre-trained language models, in particular, of Llama 3 405B.

#### 5.1.3 Adversarial Benchmarks

In addition to the benchmarks presented above, we evaluate on several adversarial benchmarks in three areas: question answering, mathematical reasoning, and paraphrase detection. This testing probes the model’s capabilities on tasks specifically created to be challenging and can potentially also point to overfitting on benchmarks. For question answering, we use Adversarial SQuAD ( Jia and Liang, 2017 ) and Dynabench SQuAD ( Kiela et al., 2021 ) . For mathematical reasoning, we use GSM-Plus ( Li et al., 2024c ) . For paraphrase detection, we use PAWS ( Zhang et al., 2019 ) .

Figure 15 presents the scores of Llama 3 8B, 70B, and 405B on the adversarial benchmarks as a function of their performance on non-adversarial benchmarks. The non-adversarial benchmarks we use are SQuAD ( Rajpurkar et al., 2016 ) for question answering, GSM8K for mathematical reasoning, and QQP ( Wang et al., 2017 ) for paraphrase detection. Each datapoint represents a pair of an adversarial and non-adversarial datasets ( e.g. QQP paired with PAWS), and we show all possible pairs within a category. The diagonal black line represents parity between adversarial and non-adversarial datasets — being on the line would indicate the model has similar performance regardless of the adversarial nature.

On paraphrase detection, neither pre-trained nor post-trained models appear to suffer from the type of adversariality with which PAWS was constructed, marking a substantial step with respect to the previous generation of models. This result confirms the findings of Weber et al. (2023a) , who also found that LLMs are less susceptible to the type of spurious correlations found in several adversarial datasets. For mathematical reasoning and question answering, however, the adversarial performances are substantially lower than the non-adversarial performances. This pattern is similar for pre-trained and post-trained models.

#### 5.1.4 Contamination Analysis

We conduct a contamination analysis to estimate to what extent benchmark scores may be influenced by contamination of the evaluation data in the pre-training corpus. In previous work, several different contamination methods have been used, with various different hyperparameters – we refer to Singh et al. (2024) for an overview. Any of these methods can suffer from false positives and negatives, and how to best run contamination analyses is currently still an open field of research. Here, we largely follow the suggestions of Singh et al. (2024) .

Method. Specifically, Singh et al. (2024) propose to select contamination detection methods empirically, based on which method results in the largest difference between the ‘clean’ part of the dataset and the entire dataset, which they call estimated performance gain . For all our evaluation datasets, we score examples based on 8-gram overlap, a method that was found by Singh et al. (2024) to be accurate for many datasets. We consider an example of a dataset D D to be contaminated if a ratio 𝒯 D \mathcal{T}_{D} of its tokens are part of an 8-gram occurring at least once in the pre-training corpus. We select 𝒯 D \mathcal{T}_{D} separately for each dataset, based on which value shows the maximal significant estimated performance gain across the three model sizes.

Results. In Table 15 , we report the percentage of evaluation data that is considered contaminated for the maximal estimated performance gain, as described above, for all key benchmarks. From the table, we exclude numbers for benchmarks for which the results are not significant, for instance because the clean or contaminated set has too few examples, or because the observed performance gain estimate shows extremely erratic behavior. In Table 15 , we observe that for some datasets contamination has a large impact, while for others it does not. For example, for PiQA and HellaSwag, both the estimation of contamination and the estimation of performance gain are high. For Natural Questions, on the other hand, the estimated 52% contamination seems to have virtually no effect on the performance. For SQuAD and MATH, low thresholds yield high levels of contamination, but no performance gains. This suggests that contamination is either not helpful for these datasets, or that a larger n is required to obtain a better estimate. Finally, for MBPP, HumanEval, MMLU and MMLU-Pro, other contamination detection methods may be needed: even with higher thresholds, 8-gram overlap gives such high contamination scores that it is impossible to get a good performance gain estimate.

### 5.2 Post-trained Language Model

We present results for our Llama 3 post-trained models on benchmarks across different capabilities. Similar to pre-training we are releasing the data generated as part of evaluations with publicly available benchmarks which can be found on Huggingface here . Additional details on our eval setup can be found here .

Benchmarks and metrics. Table 16 contains an overview of all the benchmarks, organized by the capability. We apply decontamination of the post-training data by running exact match with the prompts from each benchmark. In addition to the standard academic benchmarks, we also performed extensive human evaluation of different capabilities. Details are provided in Section 5.3 .

Experimental setup. We employ a similar experimental setup to the pre-training phase and conduct a comparative analysis of Llama 3 alongside other models of comparable size and capability. To the extent possible, we evaluate the performance of other models ourselves and compare the results with the reported numbers, selecting the best score. You can find additional details on our evaluation setup here .

#### 5.2.1 General Knowledge and Instruction-Following Benchmarks

We evaluate Llama 3 on benchmarks for general knowledge and instruction-following in Table 2 .

General knowledge. We leverage MMLU ( Hendrycks et al., 2021a ) and MMLU-Pro ( Wang et al., 2024b ) to evaluate Llama 3’s capability on knowledge-based question answering. For MMLU, we report the macro average of subtask accuracy under the 5-shot standard setting without CoT. MMLU-Pro is an extension of MMLU, incorporating more challenging, reasoning-focused questions, eliminating noisy questions, and expanding the choice set from four to ten options. Given its focus on complex reasoning, we report 5-shot CoT for MMLU-Pro. All tasks are formatted as generation tasks, similar to simple-evals ( OpenAI, 2024 ) .

As shown in Table 2 , our 8B and 70B Llama 3 variants outperform other models of similar sizes on both general knowledge tasks. Our 405B model outperforms GPT-4 and Nemotron 4 340B, with Claude 3.5 Sonnet leading among larger models.

Instruction following. We assess the ability of Llama 3 and other models to follow natural language instructions on IFEval ( Zhou et al., 2023 ) . IFEval comprises approximately 500 “verifiable instructions” such as “write in more than 400 words”, which can be verified by heuristics. We report the average of prompt-level and instruction-level accuracy, under strict and loose constraints in Table 2 . Note that all Llama 3 variants outperform comparable models across IFEval.

#### 5.2.2 Proficiency Exams

Next, we evaluate our models on a wide variety of proficiency exams originally designed to test humans. We source these exams from publicly available official sources; for some exams, we report average scores across different exam sets per proficiency exam. Specifically, we average: • GRE : Official GRE Practice Test 1 and 2 (from the Educational Testing Services);

• LSAT : Official Preptest 71, 73, 80 and 93;

• SAT : 8 exams from The Official SAT Study guide edition 2018;

• AP : One official practice exam per subject;

• GMAT Official GMAT Online Exam.