The Llama 3 Herd of Models — part 8/16

Cleaned-text code points [110218, 124740). Bibliography omitted here; source.md includes it.

Questions in these exams contain both MCQ style and generation questions. We exclude the questions that are accompanied with images. For the GRE exams that contain questions with multiple correct options, we qualify the outputs as correct only if all the correct options are selected by the model. The evaluations are run using few shot prompting wherever we have more than 1 exam set per exam. We scale the scores to be in the range 130-170 for GRE and report accuracy for all other exams.

Our results can be found in Table 17 . We observe that the performance of our Llama 3 405B model is very similar to Claude 3.5 Sonnet and GPT-4 4o. Our 70B model has an even more impressive performance. It is significantly better than GPT-3.5 Turbo and beats Nemotron 4 340B on many tests.

#### 5.2.3 Coding Benchmarks

We evaluate Llama 3 on code generation on several popular Python and multi-programming language benchmarks. To gauge the effectiveness of our models in generating functionally correct code, we use the pass@ N N metric, which evaluates the pass rate for a set of unit tests among N N generations. We report pass@1.

Python code generation. HumanEval ( Chen et al., 2021 ) and MBPP ( Austin et al., 2021 ) are popular benchmarks for Python code generation which focus on relatively simple, self-contained functions. HumanEval+ ( Liu et al., 2024a ) is an enhanced version of HumanEval, in which more tests are generated to avoid false positives. The MBPP EvalPlus base version (v0.2.0) is a selection of 378 well-formed problems out of the 974 initial problems in all of the original MBPP (train and test) dataset ( Liu et al., 2024a ) . Results for these benchmarks are reported in Table 18 . Across the Python variants of these benchmarks, Llama 3 8B and 70B outperform models of similar sizes. For the largest models, Llama 3 405B, Claude 3.5 Sonnet and GPT-4o perform similarly, with GPT-4o showing the strongest results.

Multi-programming language code generation. To assess code generation capabilities beyond Python, we report results for the MultiPL-E ( Cassano et al., 2023 ) benchmark, which is based on translations of problems from HumanEval and MBPP. Results for a subset of popular programming languages are reported in Table 19 . Note that there is a significant drop in performance compared to the Python counterparts in Table 18 .

#### 5.2.4 Multilingual Benchmarks

Llama 3 supports 8 languages --- English, German, French, Italian, Portuguese, Hindi, Spanish, and Thai, although the underlying foundation model has been trained on a broader collection of languages. 9 9 9 Llama 3 has not been optimized or safety tuned for use cases in those other languages. Developers may fine-tune Llama 3 models for languages beyond the 8 supported languages provided they comply with the Llama 3 Community License and the Acceptable Use Policy and in such cases are responsible for ensuring that any uses of Llama 3 in additional languages is done in a safe and responsible manner. In Table 20 , we show results from evaluating Llama 3 on the multilingual MMLU ( Hendrycks et al., 2021a ) and Multilingual Grade School Math (MGSM) ( Shi et al., 2022 ) benchmarks.

Multilingual MMLU. We translate MMLU questions, few-shot examples, and answers using Google Translate. We leave the task instructions in English and perform the evaluation in a 5-shot setting. In Table 20 , we report average results across German, French, Italian, Portuguese, Hindi, Spanish, and Thai.

MGSM ( Shi et al., 2022 ) . We use the same native prompts as in simple-evals ( OpenAI, 2024 ) for testing our models in a 0-shot CoT setting. In Table 20 , we report averge results across languages covered in MGSM benchmark.

We find that Llama 3 405B outperforms most other models on MGSM, achieving an average of 91.6%. On MMLU, in line with English MMLU results shown above, Llama 3 405B falls behind GPT-4o by 2%. On the other hand, both Llama 3 70B and 8B models demonstrate strong performance, leading among competitors with a wide margin on both tasks.

#### 5.2.5 Math and Reasoning Benchmarks

Our math and reasoning benchmark results are presented in Table 2 . Llama 3 8B model outperforms other models of similar sizes on GSM8K, MATH, and GPQA. Our 70B model performs significantly better than other models in its class on all the benchmarks. Finally, Llama 3 405B model is the best in its category on GSM8K and ARC-C, while on MATH, it is the second best model. On GPQA, it is competitive with GPT-4 4o, with Claude 3.5 Sonnet being the best model by a significant margin.

#### 5.2.6 Long Context Benchmarks

We consider a diverse set of tasks that span various domains and text types. In the benchmarks we list below, we focus on sub-tasks that use unbiased evaluation protocols, i.e., accuracy-based metrics rather than n-gram overlapping metrics. We also prioritize tasks that we found to be of low variance. • Needle-in-a-Haystack ( Kamradt, 2023 ) measures a model’s ability to retrieve a hidden information inserted in random parts of the long document. Our Llama 3 models demonstrate perfect needle retrieval performance, successfully retrieving 100% of needles at all document depths and context lengths. We also measure performance on Multi-needle (Table 21 ), a variation of Needle-in-a-Haystack, where we insert four needles in the context and test if a model can retrieve two of them. Our Llama 3 models achieve near perfect retrieval results.

• ZeroSCROLLS ( Shaham et al., 2023 ) is a zero-shot benchmark for natural language understanding over long texts. We report numbers on the validation set, as the ground truth answers are not publicly available. Our Llama 3 405B and 70B models either match or surpass other models on various tasks in this benchmark.

• InfiniteBench ( Zhang et al., 2024 ) requires models to understand long dependencies in the context window. We evaluate Llama 3 on En.QA (QA over novels) and En.MC (multiple-choice QA over novels), where our 405B model outperforms all others. The gains are particularly significant on En.QA.

#### 5.2.7 Tool Use Performance

We evaluate our models on a range of benchmarks for zero-shot tool use ( i.e. function calling): Nexus ( Srinivasan et al., 2023 ) , API-Bank ( Li et al., 2023b ) , Gorilla API-Bench ( Patil et al., 2023 ) , and the Berkeley Function Calling Leaderboard (BFCL) ( Yan et al., 2024 ) . Results are shown in Table 22 .

On Nexus, our Llama 3 variants perform the best compared to their counterparts. On the API-Bank, our Llama 3 8B and 70B models outperform other models in their category by a significant margin. The 405B model is behind Claude 3.5 Sonnet by only 0.6%. Finally, our 405B and 70B models perform competitively on BFCL and are close second in their respective size class. Llama 3 8B performs the best in its category.

Human evaluations. We also conduct human evaluations to test the tool use capabilities of the model, with a focus on code execution tasks. We collect 2000 user prompts related to code execution (without plotting or file uploads), plot generation, and file uploads. These prompts are collected from the LMSys dataset ( Chiang et al., 2024 ) , GAIA benchmark ( Mialon et al., 2023b ) , human annotators, and synthetic generation.

g

We compare Llama 3 405B to GPT-4o using OpenAI’s Assistants API 10 10 10 https://platform.openai.com/docs/assistants/overview . The results are provided in Figure 16 . On text-only code execution tasks and plots generation, Llama 3 405B significantly beats GPT-4o. However, it lags behind on the file upload use case.

### 5.3 Human Evaluations

In addition to evaluations on standard benchmark sets, we also perform a series of human evaluations. These evaluations allow us to measure and optimize more subtle aspects of model performance, such as our model’s tone, verbosity, and understanding of nuances and cultural contexts. Well-designed human evaluations closely reflect the user experience, providing insights into how the model performs in real-world scenarios.

Prompt collection. We collected high-quality prompt spanning a wide range of categories and difficulties. To do so, we first developed a taxonomy with categories and subcategories capturing as many model capabilities as possible. We used this taxonomy to collect about 7,000 7,000 prompts spanning six individual capabilities (English, reasoning, coding, Hindi, Spanish, and Portuguese), and three multiturn capabilities 11 11 11 For multiturn human evaluations, the number of turns is between 2 and 11 in each prompt. We assess the model response in the final turn. (English, reasoning, and coding). We ensured that within each category, prompts are uniformly distributed across subcategories. We also categorized each prompt into one of three difficulty levels and ensured that our prompt collection contains roughly 10 % 10\% easy prompts, 30 % 30\% medium prompts, and 60 % 60\% hard prompts. All the human evaluation prompt sets were subject to a thorough quality assurance process. Modeling teams did not have access to our human-evaluation prompts to prevent accidental contamination or overfitting on the test set.

Evaluation process. To perform a pairwise human evaluation of two models, we ask human annotators which of two model responses (produced by different models) they prefer. Annotators use a 7-point scale for their ratings, enabling them to indicate whether one model response is much better than, better than, slightly better than, or about the same as the other model response. When an annotator indicates that one model response is better or much better than the other model response, we consider this a “win” for that model. We perform pairwise comparisons between models in which we report win rates per capability in the prompt set.

Results. We use our human evaluation process to compare Llama 3 405B with GPT-4 (0125 API version), GPT-4o (API version), and Claude 3.5 Sonnet (API version). The results of these evaluations are presented in Figure 17 . We observe that Llama 3 405B performs approximately on par with the 0125 API version of GPT-4, while achieving mixed results (some wins and some losses) compared to GPT-4o and Claude 3.5 Sonnet. On nearly all capabilities, the win rates of Llama 3 and GPT-4 are within the margin of error. On multiturn reasoning and coding tasks, Llama 3 405B outperforms GPT-4 but it underperforms GPT-4 on multilingual (Hindi, Spanish, and Portuguese) prompts. Llama 3 performs on par with GPT-4o on English prompts, on par with Claude 3.5 Sonnet on multilingual prompts, and outperforms Claude 3.5 Sonnet on single and multiturn English prompts. However, it trails Claude 3.5 Sonnet in capabilities such as coding and reasoning. Qualitatively, we find that model performance in human evaluations is heavily influenced by nuanced factors such as model tone, response structure, and verbosity – factors that we are optimizing for in our post-training process. Overall, our human evaluation results are consistent with those on standard benchmark evaluations: Llama 3 405B is very competitive with leading industry models, making it the best-performing openly available model.

Limitations. All human evaluation results underwent a thorough data quality assurance process. However, since it is challenging to define objective criteria for evaluating model responses, human evaluations can still be influenced by personal biases, backgrounds, and preferences of human annotators, which may lead to inconsistent or unreliable results.

### 5.4 Safety

We focus our study on assessing Llama 3’s ability to generate content in a safe and responsible way, while still maximizing helpful information. Our safety work begins in the pre-training stage, primarily in the form of data cleaning and filtering. We then describe our approach to safety finetuning, focusing on how to train the model to align to specific safety policies while still retaining helpfulness. We analyze each of the Llama 3 capabilities, including multilingual, long context, tool usage, and various multimodal capabilities, to measure the effectiveness of our safety mitigations.

Subsequently, we describe our assessment of uplift for cybersecurity and chemical and biological weapons risks. Uplift refers to the additional risk introduced by new technological developments compared to using existing available technologies (such as web search).

We then describe how we leverage Red Teaming to iteratively identify and combat various safety risks across capabilities and perform a residual risk assessment.

Finally, we describe system-level safety , or the development and orchestration of classifiers around the input and output of the model itself to further enhance safety and make it easier for developers to both customize safety to various usecases and deploy generative AI in more responsible ways.

#### 5.4.1 Benchmark Construction

We create various internal benchmarks to help us develop models safely and responsibly. Our benchmarks are heavily inspired by the risk categories from the ML Commons taxonomy of hazards ( Vidgen et al., 2024 ) . While various benchmarks for language model safety exist such as ToxiGen ( Hartvigsen et al., 2022 ) , XS Test ( Röttger et al., 2023 ) , and more, a number of the new capabilities in Llama 3 do not have sufficient external benchmarks available and often external benchmarks lack sufficient coverage of breadth and depth.

For each risk category, we collect human-written prompts that are either adversarial or borderline in nature — examples of such prompts can be found in Table 23 . Adversarial prompts range from straightforward ones that directly elicit a harmful response to ones that incorporate sophisticated jailbreaking techniques. Benchmarks of these adversarial prompts are the basis for how we measure violation rate.

As a counter-metric to violation rate, we also construct false refusal benchmarks composed of borderline prompts. We refer to false refusal as occurring when a model refuses to answer in a helpful way even when a plausible, safe response is possible. Borderline prompts are prompts near the decision boundary that a well-calibrated model should handle, for example, “How can I steal the spotlight from my bestie who always acts like the main character??” Our overall benchmark size across violations and false refusals is over 4000 prompts per capability or language, and contains a mix of single-turn and multi-turn prompts.

#### 5.4.2 Safety Pre-training