The Llama 3 Herd of Models — part 14/16

Cleaned-text code points [202188, 217842). Bibliography omitted here; source.md includes it.

Llama 3 embedding. The Llama 3 embeddings are taken as the output of the 16th decoder layer. We work exclusively with the Llama 3 8B model and extract the embeddings for a given text ( i.e. written-form input text for TN or the audio transcript for PM) as if they are generated by the Llama 3 model with an empty user prompt. In a given sample, each chunk in the Llama 3 token sequence is explicitly aligned with the corresponding chunks in native input sequence for TN or PM, i.e. , TN-specific text tokens (demarcated by unicode category) or phone-rate features respectively. This allows for training the TN and PM modules with streaming input of Llama 3 tokens and embeddings.

### 8.2 Model Architecture

#### 8.2.1 Speech Understanding

On the input side, the speech module consists of two successive modules: a speech encoder and an adapter. The output of the speech module is directly fed into the language model as token representation, enabling direct interaction between speech and text tokens. Furthermore, we incorporate two new special tokens to enclose the sequence of speech representations. The speech module differs substantially from the vision module (see Section 7 ), which feeds multi-modal information into the language model via cross-attention layers. By contrast, the speech module generates embeddings that can be seamlessly integrated with text tokens, enabling the speech interface to leverage all the capabilities of the Llama 3 language model.

Speech encoder. Our speech encoder is a Conformer ( Gulati et al., 2020 ) model with 1B parameters. The input to the model consists of 80-dimensional mel-spectrogram features, which are first processed by a stride-4 stacking layer followed by a linear projection to reduce the frame length to 40 ms. The resulting features are processed by an encoder with 24 Conformer layers. Each Conformer layer has a latent dimension of 1536, and consists of two Macron-net style feed-forward networks with dimension 4096, a convolution module with kernel size 7, and a rotary attention module ( Su et al., 2024 ) with 24 attention heads.

Speech adapter. The speech adapter contains about 100M parameters. It is composed of a convolution layer, a rotary Transformer layer, and a linear layer. The convolution layer has a kernel size of 3 and a stride of 2, which is designed to reduce the speech frame length to 80ms. This allows the model to provide more coarse-grained features to the language model. The Transformer layer has a latent dimension of 3072 and a feed-forward network with a dimension of 4096 which further processes the information from speech with context after the convolutional downsampling. Finally, the linear layer maps the output dimension to match that of the language-model embedding layer.

#### 8.2.2 Speech Generation

We use Llama 3 8B embeddings in two key components for speech generation: Text Normalization and Prosody Modeling. The TN module ensures semantic correctness by contextually transforming written text into spoken form. The PM module enhances naturalness and expressiveness by predicting prosodic features using these embeddings. Together, they enable accurate and natural speech generation.

Text normalization. As a determinant of the semantic correctness of generated speech, the text normalization (TN) module carries out context-aware transformation from written-form text into the respective spoken form which is eventually verbalized by the downstream components. For example, the written-form text 123 is read as a cardinal number ( one hundred twenty three ) or spelled digit-by-digit ( one two three ) depending on the semantic context. The TN system consists of a streaming LSTM-based sequence-tagging model that predicts the sequence of handcrafted TN rules used to transform the input text ( Kang et al., 2024 ) . The neural model also takes in Llama 3 embeddings via cross attention to leverage the contextual information encoded therein, enabling minimal text token lookahead and streaming input/output.

Prosody modeling. To enhance the naturalness and expressiveness of synthesized speech, we integrate a decoder-only Transformer-based Prosody model (PM) ( Radford et al., 2021 ) that takes the Llama 3 embeddings as an additional input. This integration leverages the linguistic capabilities of Llama 3, utilizing both its textual output and intermediate embeddings at the token rate ( Devlin et al., 2018 ; Dong et al., 2019 ; Raffel et al., 2020 ; Guo et al., 2023 ) to enhance the prediction of prosody features, thus reducing the lookahead required by the model.

The PM integrates several input components to generate comprehensive prosody predictions: linguistic features derived from the text normalization front-end detailed above, tokens, and embeddings. The PM predicts three key prosodic features: log duration of each phone, log F0 (fundamental frequency) average, and log power average across the phone duration. The model comprises a uni-directional Transformer and six attention heads. Each block includes cross-attention layers and dual fully connected layers with a hidden dimension of 864. A distinctive feature of the PM is its dual cross-attention mechanism, with one layer dedicated to linguistic inputs and the other to Llama embeddings. This setup efficiently manages varying input rates without requiring explicit alignment.

### 8.3 Training Recipe

#### 8.3.1 Speech Understanding

Training of the speech module is done in two stages. The first stage, speech pre-training, leverages unlabeled data to train a speech encoder that exhibits strong generalization capabilities across languages and acoustic conditions. In the second stage, supervised fine-tuning, the adapter and pre-trained encoder are integrated with the language model, and trained jointly with it while the LLM stays frozen. This enables the model to respond to speech input. This stage uses labeled data corresponding to speech understanding abilities.

Multilingual ASR and AST modeling often results in language confusion/interference, which leads to degraded performance. A popular way to mitigate this is to incorporate language identification (LID) information, both on the source and target side. This can lead to improved performance in the predetermined set of directions, but it does come with potential loss of generality. For instance, if a translation system expects LID on both source and target side, then the model will not likely to show good zero-shot performance in directions that were not seen in training. So our challenge is to design a system that allows LID information to some extent, but keeps the model general enough such that we can have the model do speech translation in unseen directions. To address this, we design system prompts which only contain LID for the text to be emitted (target side). There is no LID information for the speech input (source side) in these prompts, which also potentially allows it to work with code-switched speech. For ASR, we use the following system prompt: Repeat after me in {language}: , where {language} comes from one of the 34 languages (English, French, etc. ) For speech translation, the system prompt is: Translate the following sentence into {language}: . This design has been shown to be effective in prompting the language model to respond in the desired language. We used the same system prompts during training and inference.

Speech pre-training. We use the self-supervised BEST-RQ algorithm ( Chiu et al., 2022 ) to pre-train the speech encoder. We apply a mask of 32-frame length with a probability of 2.5% to the input mel-spectrogram. If the speech utterances are longer than 60 seconds, we perform a random crop of 6K frames, corresponding to 60 seconds of speech. We quantize mel-spectrogram features by stacking 4 consecutive frames, projecting the 320-dimensional vectors to a 16-dimensional space, and performing a nearest-neighbor search with respect to cosine similarity metric within a codebook of 8,192 vectors. To stabilize pre-training, we employ 16 different codebooks. The projection matrix and codebooks are randomly initialized and are not updated throughout the model training. The multi-softmax loss is used only on masked frames for efficiency reasons. The encoder is trained for 500K steps with a global batch size of 2,048 utterances.

Supervised finetuning. Both the pre-trained speech encoder and the randomly initialized adapter are further jointly optimized with Llama 3 in the supervised finetuning stage. The language model remains unchanged during this process. The training data is a mixture of ASR, AST, and spoken dialogue data. The speech model for Llama 3 8B is trained for 650K updates, using a global batch size of 512 utterances and an initial learning rate of 10 − 4 10^{-4} . The speech model for Llama 3 70B is trained for 600K updates, using a global batch size of 768 utterances and an initial learning rate of 4 × 10 − 5 4\times 10^{-5} .

#### 8.3.2 Speech Generation

To support real-time processing, the prosody model employs a lookahead mechanism that considers a fixed number of future phones and a variable number of future tokens. This ensures consistent lookahead while processing incoming text, which is crucial for low-latency speech synthesis applications.

Training. We develop a dynamic alignment strategy utilizing causal masking to facilitate streamability in speech synthesis. This strategy incorporates a lookahead mechanism for a fixed number of future phones and a variable number of future tokens, aligning with the chunking process during text normalization (Section 8.1.2 ). For each phone, the token lookahead includes the maximum number of tokens defined by the chunk size, resulting in variable lookahead for Llama embeddings but fixed lookahead for phonemes.

The Llama 3 embeddings are sourced from the Llama 3 8B model, which remains frozen during the training of the Prosody Model. The input phone-rate features include both linguistic and speaker/style controllability elements. The model training is conducted with a batch size of 1,024 utterances, each with a maximum length of 500 phones. We employ a learning rate of 9 × 10 − 4 9\times 10^{-4} using the AdamW optimizer, training over 1 million updates with a learning rate warmup for the first 3,000 updates, following a cosine schedule.

Inference. During inference, the same lookahead mechanism and causal masking strategy are employed to ensure consistency between training and real-time processing. The PM handles incoming text in a streaming manner, updating the input phone by phone for phone-rate features and chunk by chunk for token-rate features. The new chunk input is updated only when the first phone for that chunk is current, maintaining the alignment and lookahead as during training.

For prosody target prediction, we employ a delayed pattern approach ( Kharitonov et al., 2021 ) , which enhances the model’s ability to capture and reproduce long-range prosodic dependencies. This approach contributes to the naturalness and expressiveness of the synthesized speech, ensuring low-latency and high-quality output.

### 8.4 Speech Understanding Results

We evaluate the speech understanding capabilities of our speech interface for Llama 3 on three tasks: (1) automatic speech recognition, (2) speech translation, and (3) spoken question answering. We compare the performance of our speech interface for Llama 3 with three state-of-the-art models for speech understanding: Whisper ( Radford et al., 2023 ) , SeamlessM4T ( Barrault et al., 2023 ) , and Gemini. 19 19 19 Due to technical limitations, we compare with the performance of Gemini on MLS reported in the original paper. In all the evaluations, we used greedy search for Llama 3 token prediction.

Speech recognition. We evaluate the ASR performance on the English datasets of Multilingual LibriSpeech (MLS; Pratap et al. (2020) ), LibriSpeech ( Panayotov et al., 2015 ) , VoxPopuli ( Wang et al., 2021a ) , and a subset of the multilingual FLEURS dataset ( Conneau et al., 2023 ) . In evaluation, the decoding results are post-processed using the Whisper text normalizer to ensure consistency in comparing with the reported results of other models. On all benchmarks, we measure the word error rate of our speech interface for Llama 3 on the standard test set of those benchmarks, except for Chinese, Japanese, Korean and Thai, where the character error rate is reported.

Table 31 shows the results of ASR evaluations. It demonstrates the strong performance of Llama 3 (and multi-modal foundation models more generally) on speech recognition tasks: our model outperforms models that are tailored to speech like Whisper 20 20 20 On FLEURS ASR, Malayalam is not officially reported for Whisper v3, so we use the average of 33 languages. and SeamlessM4T on all benchmarks. On MLS English, Llama 3 performs similarly to Gemini.

Speech translation. We also evaluate our models on speech translation tasks in which the model is asked to translate non-English speech into English text. We use the FLEURS and Covost 2 ( Wang et al., 2021b ) datasets in these evaluations, measuring BLEU scores of the translated English. Table 32 presents the results of these experiments. 21 21 21 On Covost 2, we evaluate only on 15 (out of 21) languages. The performance of our models in speech translation highlights the advantages of multimodal foundation models for tasks such as speech translation.

Spoken question answering. The speech interface of Llama 3 demonstrates remarkable question answering capabilities. The model can effortlessly comprehend code-switched speech without any prior exposure to such data. Notably, although the model was trained only on single-turn dialogue, it is capable of engaging in extended, coherent multi-turn dialogue sessions. Figure 30 presents a few examples that highlight these multilingual and multi-turn capabilities.

Safety. We evaluate the safety of our speech model on MuTox ( Costa-jussà et al., 2023 ) , a multilingual audio-based dataset of 20,000 utterances for English and Spanish and 4,000 for 19 other languages, each with toxicity labels attached. The audio is passed as input to the model and the output is evaluated for toxicity, after cleaning some special characters. We apply the MuTox classifier ( Costa-jussà et al., 2023 ) and compare the results with Gemini 1.5 Pro. We evaluate the percentage of added toxicity (AT), when the input prompt is safe and the output is toxic, and the percentage of lost toxicity (LT), when the input prompt is toxic and the answer is safe. Table 33 shows the results for English and an average across all 21 languages that we evaluated on. 22 22 22 Note that for Gemini, we encountered that a significant number of responses were empty, which could be due to safety filters on their side (though some empty responses were for non-toxic input) or to rate limits. To conduct the analysis, we assumed that all the empty responses are safe. This is the most conservative approach for results and the upper bound of what Gemini results would look like. The percentage of added toxicity is very low: our speech models have the lowest percentage of added toxicity for English, with less than 1%. It removes significantly more toxicity than it adds.

### 8.5 Speech Generation Results

For speech generation, we focus on evaluating the quality of token-wise input streaming models with the Llama 3 embeddings for the text normalization and prosody modeling tasks. The evaluation focuses on comparisons with models that do not take the Llama 3 embeddings as an additional input.