Encoder-Decoder or Decoder-Only? Revisiting Encoder-Decoder Large Language Model — part 1/2

Cleaned-text code points [0, 15271). Bibliography omitted here; source.md includes it.

# Encoder-Decoder or Decoder-Only? Revisiting Encoder-Decoder Large Language Model

###### Abstract

Recent large language model (LLM) research has undergone an architectural shift from encoder-decoder modeling to nowadays the dominant decoder-only modeling. This rapid transition, however, comes without a rigorous comparative analysis especially from the scaling perspective , raising concerns that the potential of encoder-decoder models may have been overlooked. To fill this gap, we revisit encoder-decoder LLM (RedLLM), enhancing it with recent recipes from decoder-only LLM (DecLLM). We conduct a comprehensive comparison between RedLLM, pretrained with prefix language modeling (LM), and DecLLM, pretrained with causal LM, at different model scales, ranging from ∼ \sim 150M to ∼ \sim 8B. Using RedPajama V1 (1.6T tokens) for pretraining and FLAN for instruction tuning, our experiments show that RedLLM produces compelling scaling properties and surprisingly strong performance. While DecLLM is overall more compute-optimal during pretraining, RedLLM demonstrates comparable scaling and context length extrapolation capabilities. After instruction tuning, RedLLM achieves comparable and even better results on various downstream tasks while enjoying substantially better inference efficiency. We hope our findings could inspire more efforts on re-examining RedLLM, unlocking its potential for developing powerful and efficient LLMs.

## 1 Introduction

A crucial lesson from the past decade for modeling is to design scalable and universal architectures being capable of handling different tasks (modalities) and automatically learning from massive unlabeled data, as remarked by the tremendous success of large language models (LLMs) in various research areas, particularly natural language processing ( OpenAI, 2023 ; Gemini et al., 2024 ) . Different architectures often introduce distinct inductive biases, ultimately affecting a model’s learnability, scalability, and generalization ability ( Narang et al., 2021 ) . In language modeling (LM), two architectures stood out from the literature: encoder-decoder and decoder-only . The former utilizes an encoder to process the input and a separate decoder to generate the target ( Raffel et al., 2020 ) , while the latter jointly models input understanding and target generation via a single module ( Radford et al., 2019 ) . Both architectures present unique strengths and weaknesses, leading to hot debates regarding their respective superiority ( Zhang et al., 2022a ; Wang et al., 2022 ; Fu et al., 2023 ) .

However, there has been a recent shift towards favoring decoder-only architectures for LLMs, featured by popular LLMs such as LLaMA ( Touvron et al., 2023 ) , Gemma ( Team et al., 2024 ) and Mistral ( Jiang et al., 2023 ) , with very few exceptions ( Li et al., 2023 ) . Such a shift from our understanding mostly results from the success of GPT models ( Radford et al., 2019 ; Brown et al., 2020 ) , rather than the incapability of the encoder-decoder architecture. For example, when provided with comparable compute (by setting encoder-decoder models with twice the parameter count of decoder-only models), Wang et al. (2022) showed that the encoder-decoder model significantly outperformed its decoder-only counterpart after instruction tuning; Tay et al. (2022) highlighted the importance of pretraining objective and proposed UL2, which, combined with encoder-decoder modeling, achieves superior pretraining and finetuning performance across various tasks. These findings present direct evidence about the high potential of encoder-decoder modeling, challenging the recent paradigm shift. Unfortunately, the above studies evaluate architectures solely on an “apple-to-apple” basis without accounting for their scaling property ( Kaplan et al., 2020 ) – a crucial factor in modern LLMs, which is exactly the focus of our study.

In this paper, we fill this gap by revisiting encoder-decoder LLM (RedLLM) and comparing it with decoder-only LLM (DecLLM) from the scaling perspective. We adapt recent modeling recipes from DecLLM to enhance RedLLM, especially the rotary positional embedding ( Su et al., 2024 ) with continuous positions, and pretrain RedLLM with the prefix LM objective due to its simplicity and efficiency in leveraging compute. Figure 1 shows the overview. We investigate the scaling properties of RedLLM and DecLLM by comparing their pretraining and finetuning performance on a range of tasks at various model scales, from approximately 150M to 8B parameters.

We conduct experiments by first pretraining on RedPajama V1 ( Computer, 2023 ) from scratch for roughly 1.6T tokens and then finetuning on FLAN ( Longpre et al., 2023 ) for instruction following. We perform scaling analysis on samples from RedPajama ( in-domain ) and Paloma ( Magnusson et al., 2023 , out-of-domain, ) , and evaluate the models’ zero- and few-shot capability on 13 downstream tasks. Our analysis reveals a trade-off between model quality and efficiency, with RedLLM and DecLLM each excelling in different areas. Main findings are summarized below: • RedLLM and DecLLM show similar scaling exponents, while DecLLM almost dominates the compute-optimal frontier.

• During pretraining, RedLLM performs badly at zero-shot learning; its few-shot capability scales slightly with model sizes but still lags far behind DecLLM.

• After instruction tuning, RedLLM achieves comparable zero- and few-shot performance to DecLLM across scales while enjoying significantly better inference efficiency.

• At finetuning, RedLLM benefits from the bidirectional attention in its encoder; adapting DecLLM with this structure also yields significant improvements. Still, RedLLM provides the overall best quality-efficiency trade-off.

• RedLLM also shows promising context-length extrapolation capability.

## 2 Background

Past few years have witnessed the significant progress achieved via large-scale pretraining. By learning from a large amount of unlabeled (weakly-labeled) data, neural networks could acquire massive world knowledge that is transferable to improve downstream tasks. Depending on what to transfer, previous efforts can be roughly classified into two categories: representation learning and generative learning . The former focuses on pretraining reusable modules that deliver high-quality representations being adaptable to different tasks via task-specific heads ( Simonyan & Zisserman, 2014 ; Devlin et al., 2019 ; Liu, 2019 ; Baevski et al., 2020 ; Dosovitskiy et al., 2020 ) . In contrast, the latter unifies pretraining and downstream tasks to maximize the transfer by formulating all downstream tasks as generation tasks, allowing the model to be pretrained entirely in a generative manner ( Raffel et al., 2020 ; Xue et al., 2022 ; Lewis et al., 2020 ; Zhang et al., 2022b ; Scao et al., 2022 ; Anil et al., 2023 ) . In this study, we mainly explore generative learning, as represented by LLMs.

There are two crucial factors affecting the effectiveness of LLMs: model architecture and pretraining objective . Model architecture determines the layout and structure of a neural network, directly shaping its generalization. In the literature, two architectures have been widely investigated: encoder-decoder and decoder-only, both with Transformer as the backbone ( Vaswani et al., 2017 ) . The encoder-decoder architecture decomposes the input-target mapping into three separate modules – encoder self-attention, decoder self-attention and cross-attention for modeling input-input, target-target, target-input dependency, respectively. This decomposition often improves sample and inference efficiency, making it a preferred choice for several LLMs, such as T5 ( Raffel et al., 2020 ) , BART ( Lewis et al., 2020 ) and OpenBA ( Li et al., 2023 ) . Instead, the decoder-only architecture relies on a single decoder self-attention module to handle all dependencies, unifying understanding and generation and simplifying the learning. Studies comparing both architectures are many ( Fu et al., 2023 ; Raffel et al., 2020 ; Wang et al., 2022 ) , but none of them explored the scaling landscape except ( Zhang et al., 2022a ) which nevertheless focuses on machine translation exclusively.

Pretraining objective defines what to learn and how to represent information, influencing the model’s knowledge acquisition and learning efficiency. Popular choices include span corruption that masks out random tokens from the input to form the target ( Raffel et al., 2020 ) , causal LM that simply predicts the next token (i.e., p ⁡ ( x 1 , … , x T ) = ∏ t = 1 T p ⁡ ( x t | x < t ) p(x_{1},\ldots,x_{T})=\prod_{t=1}^{T}p(x_{t}|x_{<t}) , T T is sequence length) ( Radford & Narasimhan, 2018 ) , and prefix LM that follows causal LM but is conditioned on a prefix (i.e., p ⁡ ( x k , … , x T | x < k ) = ∏ t = k T p ⁡ ( x t | x < t ) p(x_{k},\ldots,x_{T}|x_{<k})=\prod_{t=k}^{T}p(x_{t}|x_{<t}) , k k is the prefix length) ( Dong et al., 2019 ) . Tay et al. (2022) found that different objectives result in models with different capabilities and limitations, and that mixing them properly produces the best performance albeit at the cost of high implementation complexity and potentially reduced training efficiency ( Li et al., 2023 ) . We mainly employ the causal LM and prefix LM objective for the pretraining for their simplicity, leaving the exploration of other alternatives to the future.

The performance of pretrained models measured by perplexity is empirically observed following a power-law function of training data size and model size ( Kaplan et al., 2020 ; Bansal et al., 2022 ; Fernandes et al., 2023 ) , allowing us to analyze compute-optimal scaling for LLM development ( Hoffmann et al., 2022 ) . In addition, pretrained LLMs with sufficient model size often exhibits promising zero/few-shot capability, i.e., solving a task based on a few to no input-output examplers ( Brown et al., 2020 ; Gemini et al., 2024 ) . This task solving ability can be further elicited through instruction tuning, a procedure finetuning LLM on massive downstream tasks ( Sanh et al., 2022 ; Longpre et al., 2023 ; Ouyang et al., 2022 ) . We follow these practices comparing RedLLM and DecLLM under scaling, and examine their pretraining and finetuning performance with zero/few-shot evaluation.

## 3 RedLLM: Revisiting Encoder-Decoder LLM

We adopt variants of Transformer ( Vaswani et al., 2017 ) as the LLM backbone, and incorporate several recent techniques from DecLLM to improve the encoder-decoder modeling. Table 1(c) summarizes the model specifications.

For DecLLM, we employ the multi-head dot-product attention ( Vaswani et al., 2017 ) for the self-attention, SwiGLU ( Shazeer, 2020 ) as the activation for the feed-forward layer, RMSNorm ( Zhang & Sennrich, 2019 ) for normalization with the pre-normalization structure, rotary embedding ( Su et al., 2024 ) for positional encoding, and tied word embeddings. We apply extra normalization layers to query ( 𝐐 ∈ ℝ T × d h \mathbf{Q}\in\mathbb{R}^{T\times d_{h}} ), key ( 𝐊 ∈ ℝ T × d h \mathbf{K}\in\mathbb{R}^{T\times d_{h}} ), and value ( 𝐕 ∈ ℝ T × d h \mathbf{V}\in\mathbb{R}^{T\times d_{h}} ) vectors over the head dimension ( d h d_{h} ) to enhance the training stability Attn DecLLM = Softmax ​ ( LN ​ ( 𝐐 ) ​ LN ​ ( 𝐊 ) T d h ) ​ LN ​ ( 𝐕 ) , \text{Attn}_{\text{{DecLLM} }}=\text{Softmax}\left(\frac{\text{LN}(\mathbf{Q}){\text{LN}(\mathbf{K})}^{T}}{\sqrt{d_{h}}}\right)\text{LN}(\mathbf{V}), (1) where LN ​ ( ⋅ ) \text{LN}(\cdot) denotes parameter-free RMSNorm. Note because of the normalization and the scaling factor 1 / d h \nicefrac{{1}}{{\sqrt{d_{h}}}} , the attention logits are bounded into [ − d h , d h ] [-\sqrt{d_{h}},\sqrt{d_{h}}] , thus avoiding logits explosion.

RedLLM follows DecLLM. We apply the rotary embedding to all attentions, covering encoder self-attention, decoder self- and cross-attention. As shown in Figure 1(a) , we employ continuous position following DecLLM so positional information flows continuously from the encoder’s end to the decoder’s beginning. We tie all word embeddings, including encoder word embedding, decoder word embedding and output embedding. We note in experiments that RedLLM suffers more from training instability. To alleviate this problem, we add another normalization layer to attention output Attn RedLLM = LN ​ ( Attn DecLLM ) . \text{Attn}_{\text{{RedLLM} }}=\text{LN}\left(\text{Attn}_{\text{{DecLLM} }}\right). (2)

We pretrain DecLLM with the causal LM objective as this objective could fully leverage DecLLM’s capacity and aligns with the current standard. We use the prefix LM objective for RedLLM, instead. While the above specifications are functional, we acknowledge that further architectural/objective optimization is very possible for both models, which we leave to the future.

## 4 Setup

We train a set of RedLLMs and DecLLMs with model size ranging from about 150M to 8B using T5X ( Roberts et al., 2023 ) . Table 2(a) lists the model configurations, and Table 2(c) shows the pretraining and finetuning hyperparameters.

#### Pretraining

We use RedPajama V1 ( Computer, 2023 ) for pretraining, which is an open-source reproduction of the LLaMA training corpus ( Touvron et al., 2023 ) . We adopt a subword tokenizer of 32K vocabulary for tokenization. To train DecLLM, we concatenate the documents (separated by tag “[EOD]”) and then split them into sequences of 2048 tokens (i.e., T = 2048 T=2048 ) without overlapping. We further split each sequence in the middle, resulting in 1024 input and 1024 target tokens (i.e., k = 1024 k=1024 ), for training RedLLM. We use Adafactor without factorization ( Shazeer & Stern, 2018 ) for optimization. All LLMs are trained for 400K steps, totaling roughly 1.6 epochs and 1.6T tokens. Note the effective target token count for RedLLM is 0.8T, half the size used for DecLLM due to the architectural difference.

#### Finetuning

We use the FLAN collection ( Longpre et al., 2023 ) for instruction tuning, which consists of 1800+ tasks with diverse prompts. We set the maximum input/output tokens to 2048/512, and finetune models on the default data mixture (without packing). We tune all model parameters ( Zhang et al., 2024 ) , and compute the log-likelihood loss only on the target output. Depending on the model, its size and resource availability, we use up to 256/64 TPU v5p chips for pretraining/finetuning, taking up to 1 month each.

#### Evaluation

We perform scaling-law analysis for pretrained models using perplexity (PPL) on RedPajama and Paloma ( Magnusson et al., 2023 ) . RedPajama is a random subset of RedPajama V1, including 79K documents sampled from 7 sources (Arxiv, Books, C4, CC, Github, Stackexchange and Wiki). We regard RedPajama as an in-domain eval set. Paloma, instead, is independently curated for perplexity analysis, including data from 16 sources and 500+ domains. We argue its independent construction and broad domain coverage make it sufficiently distinct from our pretraining data, thus we consider evaluations using Paloma to be out-of-domain . We report averaged PPL over different sources for the evaluation. By default, we use the final checkpoint for pretraining analysis.