The Llama 3 Herd of Models — part 15/16

Cleaned-text code points [217842, 232586). Bibliography omitted here; source.md includes it.

Text normalization. To measure the effect of Llama 3 embeddings, we experimented with changing the amount of right context the model uses. We trained the model using a right context of 3 TN tokens (demarcated by unicode category). This model is compared to models that do not use the Llama 3 embeddings, using a 3-token right context or a full bi-directional context. As expected, Table 34 shows using the full right context improves performance for the model without Llama 3 embeddings. However, the model that incorporates the Llama 3 embeddings outperforms all other models, hence enabling token-rate input/output streaming without relying on long context in the input.

Prosody modeling. To evaluate the performance of the our prosody model (PM) with Llama 3 8B, we conducted two sets of human evaluation comparing models with and without Llama 3 embeddings. Raters listened to samples from different models and indicated their preferences. To generate the final speech waveform, we use an in-house transformer based acoustic model ( Wu et al., 2021 ) that predicts spectral features and a WaveRNN neural vocoder ( Kalchbrenner et al., 2018 ) to generate the final speech waveform.

First, we compare directly to a streaming baseline model without Llama 3 embeddings. In the second test, the Llama 3 8B PM is compared to a non-streaming baseline model without Llama 3 embeddings. As shown in Table 35 , the Llama 3 8B PM is preferred 60% of the time compared to the streaming baseline, and 63.6% of the time compared to the non-streaming baseline, indicating a significant improvement in perceived quality. The key advantage of the Llama 3 8B PM is its token-wise streaming capability (Section 8.2.2 ), which maintains low latency during inference. This reduces the model’s lookahead requirements, enabling more responsive and real-time speech synthesis compared to non-streaming baselines. Overall, the Llama 3 8B prosody model consistently outperforms the baseline models, demonstrating its effectiveness in enhancing the naturalness and expressiveness of synthesized speech.

## 9 Related Work

The development of Llama 3 builds on a large body of prior work studying foundation models for language, images, videos, and speech. A comprehensive overview of that work is outside the scope of this paper; we refer the reader to Bordes et al. (2024) ; Madan et al. (2024) ; Zhao et al. (2023a) for such overviews. Below, we briefly outline seminal works that directly influenced the development of Llama 3.

### 9.1 Language

Scale. Llama 3 follows the enduring trend of applying straightforward methods at ever increasing scales in foundation models. Improvements are driven by increased compute and improved data, with the 405B model using almost fifty times the pre-training compute budget of Llama 2 70B. Despite containing 405B parameters, our largest Llama 3 in fact contains fewer parameters than earlier and much less performant models such as PALM ( Chowdhery et al., 2023 ) , due to better understanding of scaling laws ( Kaplan et al., 2020 ; Hoffmann et al., 2022 ) . Little is publicly known about the size of other frontier models, such as Claude 3 or GPT 4 ( OpenAI, 2023a ) , but overall performance is compareable.

Small models. Developments in smaller models have paralleled those in large models. Models with fewer parameters can dramatically improve inference cost and simplify deployment ( Mehta et al., 2024 ; Team et al., 2024 ) . The smaller Llama 3 models achieve this by training far beyond the point of compute optimal training, effectively trading training compute for inference efficiency. An alternative path is to distill larger models into smaller ones, as in Phi ( Abdin et al., 2024 ) .

Architectures. While Llama 3 makes minimal architectural modifiations to compared to Llama 2, other recent foundation models have explored other designs. Most notably, mixture of experts architectures ( Shazeer et al., 2017 ; Lewis et al., 2021 ; Fedus et al., 2022 ; Zhou et al., 2022 ) can be used as an efficient way to increase the capacity of a models, such as in Mixtral ( Jiang et al., 2024 ) and Arctic ( Snowflake, 2024 ) . Llama 3 outperforms these models, suggesting that dense architectures are not the limiting factor, but there remain numerous trade offs in terms of training and inference efficiency, and model stability at scale.

Open source. Open weights foundation models have rapidly improved over the last year, with Llama3-405B now competitive with the current closed weight state-of-the-art. Numerous model families have recently been developed, including Mistral ( Jiang et al., 2023 ) , Falcon ( Almazrouei et al., 2023 ) , MPT ( Databricks, 2024 ) , Pythia ( Biderman et al., 2023 ) , Arctic ( Snowflake, 2024 ) , OpenELM ( Mehta et al., 2024 ) , OLMo ( Groeneveld et al., 2024 ) , StableLM ( Bellagente et al., 2024 ) , OpenLLaMA ( Geng and Liu, 2023 ) , Qwen ( Bai et al., 2023 ) , Gemma ( Team et al., 2024 ) , Grok ( XAI, 2024 ) , and Phi ( Abdin et al., 2024 ) .

Post-training. Post-training Llama 3 follows the established strategy of instruction tuning ( Chung et al., 2022 ; Ouyang et al., 2022 ) followed by alignment with human feedback ( Kaufmann et al., 2023 ) . While some studies have shown the surprising effectiveness of lightweight alignment procedures ( Zhou et al., 2024 ) , Llama 3 uses millions of human instructions and preference judgments to improve the pre-trained model, including techniques such as rejection sampling ( Bai et al., 2022 ) , supervised finetuning ( Sanh et al., 2022 ) , and Direct Preference Optimization ( Rafailov et al., 2023 ) . In order to curate these instruction and preference examples, we deploy earlier versions of Llama 3 to filter ( Liu et al., 2024c ) , re-write ( Pan et al., 2024 ) , or generate prompts and responses ( Liu et al., 2024b ) and apply these techniques through multiple rounds of post-training.

### 9.2 Multimodality

Our experiments with multimodal capabilities for Llama 3 are part of a long line of work on foundation models that jointly model multiple modalities.

Images. A substantial body of work has trained image-recognition models on large amounts of image-text pairs, for example, Mahajan et al. (2018) ; Xiao et al. (2024a) ; Team (2024) ; OpenAI (2023b) . Radford et al. (2021) presented one of the first models to jointly embed images and text via contrastive learning. More recently, a series of models has studied approaches similar to the one used in Llama 3, for example, Alayrac et al. (2022) ; Dai et al. (2023) ; Liu et al. (2023c) ; Liu et al. (2023b) ; Yang et al. (2023b) ; Ye et al. (2023) ; Zhu et al. (2023) . Our approach in Llama 3 combines ideas from many of these papers to achieve results that are comparable with Gemini 1.0 Ultra ( Google, 2023 ) and GPT-4 Vision ( OpenAI, 2023b ) ; see Section 7.6 .

Video. Although video inputs are supported by an increasing number of foundation models ( Google, 2023 ; OpenAI, 2023b ) , the body of work on joint modeling of videos and language is not that large. Akin to Llama 3, most current studies adopt an adapter approach to align video and language representations and unlock question-answering and reasoning about videos ( Lin et al., 2023 ; Li et al., 2023a ; Maaz et al., 2024 ; Zhang et al., 2023 ; Zhao et al., 2022 ) . We find that such approaches produce results that are competitive with the state-of-the-art; see Section 7.7 .

Speech. Our work also fits in a larger body of work combining language and speech modeling. Earlier joint models of text and speech include AudioPaLM ( Rubenstein et al., 2023 ) , VioLA ( Wang et al., 2023b ) , VoxtLM Maiti et al. (2023) , SUTLM ( Chou et al., 2023 ) , and Spirit-LM ( Nguyen et al., 2024 ) . Our work builds on prior compositional approaches to combining speech and language like Fathullah et al. (2024) . Unlike most prior work, we opt to not finetune the language model itself for speech tasks as doing so may lead to contention on non-speech tasks. We find that at larger model scales, strong performances are attainable even without such finetuning; see Section 8.4 .

## 10 Conclusion

In many ways, the development of high-quality foundation models is still in its infancy. Our experience in developing Llama 3 suggests that substantial further improvements of these models are on the horizon. Throughout the development of the Llama 3 model family, we found that a strong focus on high-quality data, scale, and simplicity consistently yielded the best results. In preliminary experiments, we explored more complex model architectures and training recipes but did not find the benefits of such approaches to outweigh the additional complexity they introduce in model development.

Developing a flagship foundation model such as Llama 3 involves overcoming a plethora of deep technical problems but also requires clever organizational decisions. For example, to ensure Llama 3 is not accidentally overfitted on commonly used benchmarks, our pre-training data was procured and processed by a separate team that was strongly incentivized to prevent contamination of that pre-training data with external benchmarks. As another example, we ensure that our human evaluations remain trustworthy by allowing only a small set of researchers who do not contribute to model development to perform and access these evaluations. While such organizational decisions are rarely discussed in technical papers, we found them to be pivotal to the successful development of the Llama 3 family of models.

We shared the details of our development process because we believe this will: (1) help the larger research community understand the key factors of foundation model development and (2) contribute to a more informed debate about the future of foundation models in the general public. We also shared preliminary experiments with integrating multimodal capabilities into Llama 3. While these models are still under active development and not yet ready for release, we hope sharing our results early will accelerate research in this direction.

Following the positive outcomes of the detailed safety analyses presented in this paper, we publicly release our Llama 3 language models in order to accelerate the development of AI systems for a plethora of societally relevant use cases and enable the research community to scrutinize our models and identify ways to make these models better and safer. We believe that the public release of foundation models plays a key role in the responsible development of such models, and we hope that the release of Llama 3 encourages the industry to embrace the open, responsible development of AGI.

## Contributors and Acknowledgements

Llama 3 is the result of the work of a large number of people at Meta. Below, we list all core contributors (people who worked on Llama 3 for at least 2 / 3 \nicefrac{{2}}{{3}} rd of the runtime of the project) and contributors (people who worked on Llama 3 for at least 1 / 5 \nicefrac{{1}}{{5}} th of the runtime of the project). We list all contributors in alphabetical order of first name.

### Core Contributors

Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, Amy Yang, Angela Fan, Anirudh Goyal, Anthony Hartshorn, Aobo Yang, Archi Mitra, Archie Sravankumar, Artem Korenev, Arthur Hinsvark, Arun Rao, Aston Zhang, Aurelien Rodriguez, Austen Gregerson, Ava Spataru, Baptiste Roziere, Bethany Biron, Binh Tang, Bobbie Chern, Charlotte Caucheteux, Chaya Nayak, Chloe Bi, Chris Marra, Chris McConnell, Christian Keller, Christophe Touret, Chunyang Wu, Corinne Wong, Cristian Canton Ferrer, Cyrus Nikolaidis, Damien Allonsius, Daniel Song, Danielle Pintz, Danny Livshits, Danny Wyatt, David Esiobu, Dhruv Choudhary, Dhruv Mahajan, Diego Garcia-Olano, Diego Perino, Dieuwke Hupkes, Egor Lakomkin, Ehab AlBadawy, Elina Lobanova, Emily Dinan, Eric Michael Smith, Filip Radenovic, Francisco Guzmán, Frank Zhang, Gabriel Synnaeve, Gabrielle Lee, Georgia Lewis Anderson, Govind Thattai, Graeme Nail, Gregoire Mialon, Guan Pang, Guillem Cucurell, Hailey Nguyen, Hannah Korevaar, Hu Xu, Hugo Touvron, Iliyan Zarov, Imanol Arrieta Ibarra, Isabel Kloumann, Ishan Misra, Ivan Evtimov, Jack Zhang, Jade Copet, Jaewon Lee, Jan Geffert, Jana Vranes, Jason Park, Jay Mahadeokar, Jeet Shah, Jelmer van der Linde, Jennifer Billock, Jenny Hong, Jenya Lee, Jeremy Fu, Jianfeng Chi, Jianyu Huang, Jiawen Liu, Jie Wang, Jiecao Yu, Joanna Bitton, Joe Spisak, Jongsoo Park, Joseph Rocca, Joshua Johnstun, Joshua Saxe, Junteng Jia, Kalyan Vasuden Alwala, Karthik Prasad, Kartikeya Upasani, Kate Plawiak, Ke Li, Kenneth Heafield, Kevin Stone, Khalid El-Arini, Krithika Iyer, Kshitiz Malik, Kuenley Chiu, Kunal Bhalla, Kushal Lakhotia, Lauren Rantala-Yeary, Laurens van der Maaten, Lawrence Chen, Liang Tan, Liz Jenkins, Louis Martin, Lovish Madaan, Lubo Malo, Lukas Blecher, Lukas Landzaat, Luke de Oliveira, Madeline Muzzi, Mahesh Pasupuleti, Mannat Singh, Manohar Paluri, Marcin Kardas, Maria Tsimpoukelli, Mathew Oldham, Mathieu Rita, Maya Pavlova, Melanie Kambadur, Mike Lewis, Min Si, Mitesh Kumar Singh, Mona Hassan, Naman Goyal, Narjes Torabi, Nikolay Bashlykov, Nikolay Bogoychev, Niladri Chatterji, Ning Zhang, Olivier Duchenne, Onur Çelebi, Patrick Alrassy, Pengchuan Zhang, Pengwei Li, Petar Vasic, Peter Weng, Prajjwal Bhargava, Pratik Dubal, Praveen Krishnan, Punit Singh Koura, Puxin Xu, Qing He, Qingxiao Dong, Ragavan Srinivasan, Raj Ganapathy, Ramon Calderer, Ricardo Silveira Cabral, Robert Stojnic, Roberta Raileanu, Rohan Maheswari, Rohit Girdhar, Rohit Patel, Romain Sauvestre, Ronnie Polidoro, Roshan Sumbaly, Ross Taylor, Ruan Silva, Rui Hou, Rui Wang, Saghar Hosseini, Sahana Chennabasappa, Sanjay Singh, Sean Bell, Seohyun Sonia Kim, Sergey Edunov, Shaoliang Nie, Sharan Narang, Sharath Raparthy, Sheng Shen, Shengye Wan, Shruti Bhosale, Shun Zhang, Simon Vandenhende, Soumya Batra, Spencer Whitman, Sten Sootla, Stephane Collot, Suchin Gururangan, Sydney Borodinsky, Tamar Herman, Tara Fowler, Tarek Sheasha, Thomas Georgiou, Thomas Scialom, Tobias Speckbacher, Todor Mihaylov, Tong Xiao, Ujjwal Karn, Vedanuj Goswami, Vibhor Gupta, Vignesh Ramanathan, Viktor Kerkez, Vincent Gonguet, Virginie Do, Vish Vogeti, Vítor Albiero, Vladan Petrovic, Weiwei Chu, Wenhan Xiong, Wenyin Fu, Whitney Meers, Xavier Martinet, Xiaodong Wang, Xiaofang Wang, Xiaoqing Ellen Tan, Xide Xia, Xinfeng Xie, Xuchao Jia, Xuewei Wang, Yaelle Goldschlag, Yashesh Gaur, Yasmine Babaei, Yi Wen, Yiwen Song, Yuchen Zhang, Yue Li, Yuning Mao, Zacharie Delpierre Coudert, Zheng Yan, Zhengxing Chen, and Zoe Papakipos.

### Contributors