# Encoder-Decoder or Decoder-Only? Revisiting Encoder-Decoder Large Language Model

###### Abstract

Recent large language model (LLM) research has undergone an architectural shift from encoder-decoder modeling to nowadays the dominant decoder-only modeling. This rapid transition, however, comes without a rigorous comparative analysis especially from the scaling perspective , raising concerns that the potential of encoder-decoder models may have been overlooked. To fill this gap, we revisit encoder-decoder LLM (RedLLM), enhancing it with recent recipes from decoder-only LLM (DecLLM). We conduct a comprehensive comparison between RedLLM, pretrained with prefix language modeling (LM), and DecLLM, pretrained with causal LM, at different model scales, ranging from ∼ \sim 150M to ∼ \sim 8B. Using RedPajama V1 (1.6T tokens) for pretraining and FLAN for instruction tuning, our experiments show that RedLLM produces compelling scaling properties and surprisingly strong performance. While DecLLM is overall more compute-optimal during pretraining, RedLLM demonstrates comparable scaling and context length extrapolation capabilities. After instruction tuning, RedLLM achieves comparable and even better results on various downstream tasks while enjoying substantially better inference efficiency. We hope our findings could inspire more efforts on re-examining RedLLM, unlocking its potential for developing powerful and efficient LLMs.

## 1 Introduction

A crucial lesson from the past decade for modeling is to design scalable and universal architectures being capable of handling different tasks (modalities) and automatically learning from massive unlabeled data, as remarked by the tremendous success of large language models (LLMs) in various research areas, particularly natural language processing ( OpenAI, 2023 ; Gemini et al., 2024 ) . Different architectures often introduce distinct inductive biases, ultimately affecting a model’s learnability, scalability, and generalization ability ( Narang et al., 2021 ) . In language modeling (LM), two architectures stood out from the literature: encoder-decoder and decoder-only . The former utilizes an encoder to process the input and a separate decoder to generate the target ( Raffel et al., 2020 ) , while the latter jointly models input understanding and target generation via a single module ( Radford et al., 2019 ) . Both architectures present unique strengths and weaknesses, leading to hot debates regarding their respective superiority ( Zhang et al., 2022a ; Wang et al., 2022 ; Fu et al., 2023 ) .

However, there has been a recent shift towards favoring decoder-only architectures for LLMs, featured by popular LLMs such as LLaMA ( Touvron et al., 2023 ) , Gemma ( Team et al., 2024 ) and Mistral ( Jiang et al., 2023 ) , with very few exceptions ( Li et al., 2023 ) . Such a shift from our understanding mostly results from the success of GPT models ( Radford et al., 2019 ; Brown et al., 2020 ) , rather than the incapability of the encoder-decoder architecture. For example, when provided with comparable compute (by setting encoder-decoder models with twice the parameter count of decoder-only models), Wang et al. (2022) showed that the encoder-decoder model significantly outperformed its decoder-only counterpart after instruction tuning; Tay et al. (2022) highlighted the importance of pretraining objective and proposed UL2, which, combined with encoder-decoder modeling, achieves superior pretraining and finetuning performance across various tasks. These findings present direct evidence about the high potential of encoder-decoder modeling, challenging the recent paradigm shift. Unfortunately, the above studies evaluate architectures solely on an “apple-to-apple” basis without accounting for their scaling property ( Kaplan et al., 2020 ) – a crucial factor in modern LLMs, which is exactly the focus of our study.

In this paper, we fill this gap by revisiting encoder-decoder LLM (RedLLM) and comparing it with decoder-only LLM (DecLLM) from the scaling perspective. We adapt recent modeling recipes from DecLLM to enhance RedLLM, especially the rotary positional embedding ( Su et al., 2024 ) with continuous positions, and pretrain RedLLM with the prefix LM objective due to its simplicity and efficiency in leveraging compute. Figure 1 shows the overview. We investigate the scaling properties of RedLLM and DecLLM by comparing their pretraining and finetuning performance on a range of tasks at various model scales, from approximately 150M to 8B parameters.

We conduct experiments by first pretraining on RedPajama V1 ( Computer, 2023 ) from scratch for roughly 1.6T tokens and then finetuning on FLAN ( Longpre et al., 2023 ) for instruction following. We perform scaling analysis on samples from RedPajama ( in-domain ) and Paloma ( Magnusson et al., 2023 , out-of-domain, ) , and evaluate the models’ zero- and few-shot capability on 13 downstream tasks. Our analysis reveals a trade-off between model quality and efficiency, with RedLLM and DecLLM each excelling in different areas. Main findings are summarized below: • RedLLM and DecLLM show similar scaling exponents, while DecLLM almost dominates the compute-optimal frontier.

• During pretraining, RedLLM performs badly at zero-shot learning; its few-shot capability scales slightly with model sizes but still lags far behind DecLLM.

• After instruction tuning, RedLLM achieves comparable zero- and few-shot performance to DecLLM across scales while enjoying significantly better inference efficiency.

• At finetuning, RedLLM benefits from the bidirectional attention in its encoder; adapting DecLLM with this structure also yields significant improvements. Still, RedLLM provides the overall best quality-efficiency trade-off.

• RedLLM also shows promising context-length extrapolation capability.

## 2 Background

Past few years have witnessed the significant progress achieved via large-scale pretraining. By learning from a large amount of unlabeled (weakly-labeled) data, neural networks could acquire massive world knowledge that is transferable to improve downstream tasks. Depending on what to transfer, previous efforts can be roughly classified into two categories: representation learning and generative learning . The former focuses on pretraining reusable modules that deliver high-quality representations being adaptable to different tasks via task-specific heads ( Simonyan & Zisserman, 2014 ; Devlin et al., 2019 ; Liu, 2019 ; Baevski et al., 2020 ; Dosovitskiy et al., 2020 ) . In contrast, the latter unifies pretraining and downstream tasks to maximize the transfer by formulating all downstream tasks as generation tasks, allowing the model to be pretrained entirely in a generative manner ( Raffel et al., 2020 ; Xue et al., 2022 ; Lewis et al., 2020 ; Zhang et al., 2022b ; Scao et al., 2022 ; Anil et al., 2023 ) . In this study, we mainly explore generative learning, as represented by LLMs.

There are two crucial factors affecting the effectiveness of LLMs: model architecture and pretraining objective . Model architecture determines the layout and structure of a neural network, directly shaping its generalization. In the literature, two architectures have been widely investigated: encoder-decoder and decoder-only, both with Transformer as the backbone ( Vaswani et al., 2017 ) . The encoder-decoder architecture decomposes the input-target mapping into three separate modules – encoder self-attention, decoder self-attention and cross-attention for modeling input-input, target-target, target-input dependency, respectively. This decomposition often improves sample and inference efficiency, making it a preferred choice for several LLMs, such as T5 ( Raffel et al., 2020 ) , BART ( Lewis et al., 2020 ) and OpenBA ( Li et al., 2023 ) . Instead, the decoder-only architecture relies on a single decoder self-attention module to handle all dependencies, unifying understanding and generation and simplifying the learning. Studies comparing both architectures are many ( Fu et al., 2023 ; Raffel et al., 2020 ; Wang et al., 2022 ) , but none of them explored the scaling landscape except ( Zhang et al., 2022a ) which nevertheless focuses on machine translation exclusively.

Pretraining objective defines what to learn and how to represent information, influencing the model’s knowledge acquisition and learning efficiency. Popular choices include span corruption that masks out random tokens from the input to form the target ( Raffel et al., 2020 ) , causal LM that simply predicts the next token (i.e., p ⁡ ( x 1 , … , x T ) = ∏ t = 1 T p ⁡ ( x t | x < t ) p(x_{1},\ldots,x_{T})=\prod_{t=1}^{T}p(x_{t}|x_{<t}) , T T is sequence length) ( Radford & Narasimhan, 2018 ) , and prefix LM that follows causal LM but is conditioned on a prefix (i.e., p ⁡ ( x k , … , x T | x < k ) = ∏ t = k T p ⁡ ( x t | x < t ) p(x_{k},\ldots,x_{T}|x_{<k})=\prod_{t=k}^{T}p(x_{t}|x_{<t}) , k k is the prefix length) ( Dong et al., 2019 ) . Tay et al. (2022) found that different objectives result in models with different capabilities and limitations, and that mixing them properly produces the best performance albeit at the cost of high implementation complexity and potentially reduced training efficiency ( Li et al., 2023 ) . We mainly employ the causal LM and prefix LM objective for the pretraining for their simplicity, leaving the exploration of other alternatives to the future.

The performance of pretrained models measured by perplexity is empirically observed following a power-law function of training data size and model size ( Kaplan et al., 2020 ; Bansal et al., 2022 ; Fernandes et al., 2023 ) , allowing us to analyze compute-optimal scaling for LLM development ( Hoffmann et al., 2022 ) . In addition, pretrained LLMs with sufficient model size often exhibits promising zero/few-shot capability, i.e., solving a task based on a few to no input-output examplers ( Brown et al., 2020 ; Gemini et al., 2024 ) . This task solving ability can be further elicited through instruction tuning, a procedure finetuning LLM on massive downstream tasks ( Sanh et al., 2022 ; Longpre et al., 2023 ; Ouyang et al., 2022 ) . We follow these practices comparing RedLLM and DecLLM under scaling, and examine their pretraining and finetuning performance with zero/few-shot evaluation.

## 3 RedLLM: Revisiting Encoder-Decoder LLM

We adopt variants of Transformer ( Vaswani et al., 2017 ) as the LLM backbone, and incorporate several recent techniques from DecLLM to improve the encoder-decoder modeling. Table 1(c) summarizes the model specifications.

For DecLLM, we employ the multi-head dot-product attention ( Vaswani et al., 2017 ) for the self-attention, SwiGLU ( Shazeer, 2020 ) as the activation for the feed-forward layer, RMSNorm ( Zhang & Sennrich, 2019 ) for normalization with the pre-normalization structure, rotary embedding ( Su et al., 2024 ) for positional encoding, and tied word embeddings. We apply extra normalization layers to query ( 𝐐 ∈ ℝ T × d h \mathbf{Q}\in\mathbb{R}^{T\times d_{h}} ), key ( 𝐊 ∈ ℝ T × d h \mathbf{K}\in\mathbb{R}^{T\times d_{h}} ), and value ( 𝐕 ∈ ℝ T × d h \mathbf{V}\in\mathbb{R}^{T\times d_{h}} ) vectors over the head dimension ( d h d_{h} ) to enhance the training stability Attn DecLLM = Softmax ​ ( LN ​ ( 𝐐 ) ​ LN ​ ( 𝐊 ) T d h ) ​ LN ​ ( 𝐕 ) , \text{Attn}_{\text{{DecLLM} }}=\text{Softmax}\left(\frac{\text{LN}(\mathbf{Q}){\text{LN}(\mathbf{K})}^{T}}{\sqrt{d_{h}}}\right)\text{LN}(\mathbf{V}), (1) where LN ​ ( ⋅ ) \text{LN}(\cdot) denotes parameter-free RMSNorm. Note because of the normalization and the scaling factor 1 / d h \nicefrac{{1}}{{\sqrt{d_{h}}}} , the attention logits are bounded into [ − d h , d h ] [-\sqrt{d_{h}},\sqrt{d_{h}}] , thus avoiding logits explosion.

RedLLM follows DecLLM. We apply the rotary embedding to all attentions, covering encoder self-attention, decoder self- and cross-attention. As shown in Figure 1(a) , we employ continuous position following DecLLM so positional information flows continuously from the encoder’s end to the decoder’s beginning. We tie all word embeddings, including encoder word embedding, decoder word embedding and output embedding. We note in experiments that RedLLM suffers more from training instability. To alleviate this problem, we add another normalization layer to attention output Attn RedLLM = LN ​ ( Attn DecLLM ) . \text{Attn}_{\text{{RedLLM} }}=\text{LN}\left(\text{Attn}_{\text{{DecLLM} }}\right). (2)

We pretrain DecLLM with the causal LM objective as this objective could fully leverage DecLLM’s capacity and aligns with the current standard. We use the prefix LM objective for RedLLM, instead. While the above specifications are functional, we acknowledge that further architectural/objective optimization is very possible for both models, which we leave to the future.

## 4 Setup

We train a set of RedLLMs and DecLLMs with model size ranging from about 150M to 8B using T5X ( Roberts et al., 2023 ) . Table 2(a) lists the model configurations, and Table 2(c) shows the pretraining and finetuning hyperparameters.

#### Pretraining

We use RedPajama V1 ( Computer, 2023 ) for pretraining, which is an open-source reproduction of the LLaMA training corpus ( Touvron et al., 2023 ) . We adopt a subword tokenizer of 32K vocabulary for tokenization. To train DecLLM, we concatenate the documents (separated by tag “[EOD]”) and then split them into sequences of 2048 tokens (i.e., T = 2048 T=2048 ) without overlapping. We further split each sequence in the middle, resulting in 1024 input and 1024 target tokens (i.e., k = 1024 k=1024 ), for training RedLLM. We use Adafactor without factorization ( Shazeer & Stern, 2018 ) for optimization. All LLMs are trained for 400K steps, totaling roughly 1.6 epochs and 1.6T tokens. Note the effective target token count for RedLLM is 0.8T, half the size used for DecLLM due to the architectural difference.

#### Finetuning

We use the FLAN collection ( Longpre et al., 2023 ) for instruction tuning, which consists of 1800+ tasks with diverse prompts. We set the maximum input/output tokens to 2048/512, and finetune models on the default data mixture (without packing). We tune all model parameters ( Zhang et al., 2024 ) , and compute the log-likelihood loss only on the target output. Depending on the model, its size and resource availability, we use up to 256/64 TPU v5p chips for pretraining/finetuning, taking up to 1 month each.

#### Evaluation

We perform scaling-law analysis for pretrained models using perplexity (PPL) on RedPajama and Paloma ( Magnusson et al., 2023 ) . RedPajama is a random subset of RedPajama V1, including 79K documents sampled from 7 sources (Arxiv, Books, C4, CC, Github, Stackexchange and Wiki). We regard RedPajama as an in-domain eval set. Paloma, instead, is independently curated for perplexity analysis, including data from 16 sources and 500+ domains. We argue its independent construction and broad domain coverage make it sufficiently distinct from our pretraining data, thus we consider evaluations using Paloma to be out-of-domain . We report averaged PPL over different sources for the evaluation. By default, we use the final checkpoint for pretraining analysis.

We also evaluate zero/few-shot performance of pretrained and finetuned models. We follow Longpre et al. (2023) and extend the evaluation to 13 tasks: BoolQ ( Clark et al., 2019 ) , HellaSwag ( Zellers et al., 2019 ) , RTE ( Bentivogli et al., 2009 ) , ARC easy and challenge ( Clark et al., 2018 ) , ANLI R1, R2 and R3 ( Nie et al., 2020 ) , CommonsenseQA ( Talmor et al., 2019 ) , StrategyQA ( Geva et al., 2021 ) , TriviaQA ( Joshi et al., 2017 ) , UnifiedQA ( Khashabi et al., 2020 ) , MMLU ( Hendrycks et al., 2021 , CoT) , GSM8K ( Cobbe et al., 2021 ) , BIG-Bench Hard ( bench authors, 2023 , BBH CoT,) , and WMT ( Kocmi et al., 2022 , English ↔ \leftrightarrow German, English ↔ \leftrightarrow Chinese,) . We report accuracy except WMT for which we use ChrF. We employ 5 shots for few-shot evaluation except BBH (3 shots), GSM8K (8 shots), CommonsenseQA (7 shots), and StrategyQA (6 shots). We adopt greedy decoding for all models. We calculate averaged score over all tasks for model comparison, and report the best results for finetuning.

## 5 Pretraining Results

#### RedLLM and DecLLM scale at similar rates when increasing the compute (training Flops) and model parameters.

The substantial differences in architecture and pretraining objective make it challenging to find a common ground comparing RedLLM and DecLLM. Doubling model parameters for RedLLM as in ( Wang et al., 2022 ; Tay et al., 2022 ) is arguably unfair. We instead resort to scaling law ( Kaplan et al., 2020 ) and compare them under two factors: model parameters and training Flops. Figure 2 and 10 show the results; Table 2(b) summarizes the scaling exponents. We also show the pretraining loss curve in Figure 11 .

Interestingly, RedLLM shows comparable scaling capability to DecLLM regardless of evaluation domains, as demonstrated by similar scaling exponents. DecLLM is more parameter efficient, outperforming RedLLM by a consistent gap under the same amount of parameters. However, this comes at the cost of computational inefficiency, as it requires approximately double the Flops to train under similar conditions. When shifting the comparison basis to compute, the quality gap almost disappears where both scaling curves nearly overlap.

#### RedLLM lags behind DecLLM for compute-optimal training.

We next investigate how effectively different models utilize the compute, as measured by PPL during the pretraining. Intuitively, a model achieving lower PPL under the same compute budget is preferred. Figure 3 and 13 show the comparison between RedLLM and DecLLM with different compute budgets. We first plot the PPL-to-compute curve for all models, where the bottom left region indicates the Pareto frontier. We then use this information to fit a compute-optimal scaling law function ( Hoffmann et al., 2022 ) for both RedLLM and DecLLM, and plot the isoFLOP for both. Figure 3 (and 13 ) shows that while RedLLM shows a slight edge in low-compute settings, DecLLM’s advantage becomes dominant as the compute scales up. This superior scaling behavior might come from its causal LM objective, which endows DecLLM with higher efficiency in utilizing training tokens. Still, the quality gap between RedLLM and DecLLM is not that substantial and narrows with the increase of compute.

#### RedLLM scales slightly for few-shot learning but is poor at zero-shot; both largely underperform DecLLM.

While PPL has been widely used for measuring LLM capability, it may not be indicative enough for the downstream task-solving ability ( Kuribayashi et al., 2021 ; Hu et al., 2024 ) . Apart from PPL, we further examine models’ zero/few-shot performance as shown in Figure 4 . RedLLM and DecLLM both achieve non-trivial few-shot performance, demonstrating their capability of in-context learning. DecLLM yields consistently better performance as scaling up model and data size, for both zero-shot and few-shot learning. In contrast, RedLLM performs badly at zero-shot learning, and only shows slight benefits from the scaling for few-shot learning. Its inferior zero-shot results, nevertheless, may not result from RedLLM’s insufficient capability but from an inadequate way of zero-shot prompting ( Patel et al., 2023 ) . In general, RedLLM falls behind DecLLM substantially at zero/few-shot learning during pretraining, resonating with the findings of Wang et al. (2022) . This still holds true even when their PPL scores are comparable, e.g., RedLLM 8B vs. DecLLM 4B.

#### RedLLM shows comparable and even better length extrapolation capability than DecLLM along model scaling.

One strength of DecLLM with rotary embedding is its length extrapolation ability, i.e., being able to handle sequences beyond pretraining context length. This raises the question of how RedLLM performs on long sequence modeling and we show the results in Figure 5 and 12 . In general, PPL goes up – becomes worse – for both RedLLM and DecLLM as context length increases, regardless of prefix lengths and domains.

DecLLM maintains relatively stable PPL up to a context length of 4096 tokens, i.e., twice the pretraining length. Beyond this point, PPL starts increasing, eventually exhibiting divergent behavior. We note that the larger DecLLM is, the more rapid PPL escalates as sequence length extends. In contrast, RedLLM shows a surprisingly smoother increase in PPL, regardless of model sizes, demonstrating a stronger ability to extrapolate to longer sequences. Nevertheless, it often underperforms DecLLM on sequences shorter than 4K tokens.

Interestingly, the impact of prefix length is marginal on the extrapolation behavior except RedLLM at prefix length of 1. RedLLM suffers from significant quality drop on short sequences when reducing the prefix length to 1 since the capacity from its encoder becomes limited. As model scales up, however, the gap gets largely narrowed, perhaps due to the improved decoder capacity.

#### The decoder self- and cross-attention in RedLLM show intriguing patterns under long context.

The above results motivate us to dive deeper into the models for a better understanding of what happened under long context. We find from Figure 6 (top) that the groundtruth target token’s log probability often decreases as token position increases. In contrast to RedLLM, DecLLM tends to suffer from a sharp probability drop after some point. We further inspect the decoder self-attention with the suspect that self-attention weights might not be functioning as intended. Figure 7(b) and 7(c) show that the decoder self-attention in RedLLM and DecLLM follows a clear “ locality ” structure where tokens favor attending to itself and nearby tokens, but the strength of locality weakens with the increase of token position particularly in DecLLM– a phenomenon we called locality decay . To verify this finding, we calculated the local attention probability, i.e., summed attention weights over a local window size of 5. Figure 6 (bottom) shows that the decay behavior occurs in both RedLLM and DecLLM, while DecLLM suffers more.

Figure 7(a) shows that the pattern of cross-attention weights differs significantly from that of self-attention weights. There is always a subset of input tokens attended by target tokens in RedLLM, and this subset varies little ( Zhang et al., 2020 ) . Such diverse attention may help RedLLM to capture some distant information under long context modeling, but may also increase learning difficulty as the input-target dependency in language modeling is of higher uncertainty. Still, the underlying reason behind RedLLM’s long context behavior remains unclear, which we leave to the future.

## 6 Finetuning Results

#### RedLLM shows high adaptability: matching and even surpassing DecLLM across scales after finetuning.

While RedLLM’s zero/few-shot performance lags behind DecLLM in pretraining, finetuning changes the situation. Figure 8 shows the averaged scaling results for finetuning along with pretraining under model parameters and inference Flops, and Table 3 lists detailed numbers. RedLLM catches up with DecLLM after finetuning, even under the same amount of model parameters (where RedLLM produces higher PPL than DecLLM). When switching the comparison basis to the inference Flops, RedLLM shows significant advantage, nearly dominating the quality-compute Pareto frontier. The performance gap between zero-shot and few-shot evaluation is also largely reduced. Apart from averaged results, we add per-task performance in Figures 14 , 15 and 16 . The results vary across tasks, e.g., RedLLM largely surpasses DecLLM on ANLI, while DecLLM excels at ARC. These differences should be carefully considered when comparing both models.

Note Wang et al. (2022) also observed similar results. However, their experiments were based on a 11B encoder-decoder model and 4.8B decoder-only model. In contrast, we present the scaling landscape across multiple model sizes. We further demonstrate the feasibility of achieving comparable performance to DecLLM under similar model sizes.

In addition, these results suggest that the adaptability of different models varies greatly. Models with worse pretraining performance may stand out after finetuning, such as RedLLM. We conjecture that the superior pretraining performance of DecLLM is mostly caused by the higher degree of matching between its pretraining objective and the downstream evaluation, rather than its stronger capability. Surprisingly, PPL also fails to capture a model’s adaptability, echoing with Hu et al. (2024) .

#### Bidirectional input attention improves DecLLM greatly but doesn’t change the quality-compute frontier.

An important difference between RedLLM and DecLLM at finetuning is the bidirectional input attention (BiAttn) in RedLLM’s encoder, which allows input tokens attending to each other. Such a structure often improves input understanding, and has proven useful in various settings including DecLLM ( Wang et al., 2022 ; Zhang et al., 2022a ) . This may explain why DecLLM underperforms RedLLM after finetuning.

We thus enable this structure for DecLLM. Figure 8 and Table 3 show that the performance of DecLLM gets substantially improved. Interestingly, while DecLLM + BiAttn largely outperforms RedLLM at zero-shot learning, their few-shot quality gap becomes smaller to negligible. Still, the overall quality gap among different models is much smaller compared to the pretraining performance gap, and the leading position of RedLLM with respect to the quality-compute trade-off remains.

#### RedLLM has clear advantage over DecLLM on training and inference efficiency.

Beyond theoretical Flops analysis, our empirical efficiency analysis (Figure 9 ) reveals a striking advantage for RedLLM in both training and inference throughput. This advantage becomes even more pronounced as model size increases, particularly during pretraining, where RedLLM significantly outpaces DecLLM. This remarkable efficiency, together with RedLLM’s promising finetuning performance, underscores its significance, particularly on the fact that instruction tuning and alignment have become an essential part of LLM development ( Ouyang et al., 2022 ) .

## 7 Conclusion and Future Work

Encoder-decoder or decoder-only for LLM? We answered this question from the scaling perspective and provided a comprehensive analysis over multiple dimensions. We conducted large-scale pretraining and finetuning experiments using RedPajama V1 and FLAN, with models ranging from 150M to 8B parameters. Our results show distinct strengths for each architecture. DecLLM presents unique strengths during pretraining, including compute-optimal scaling, zero/few-shot learning, and strong training stability. By contrast, RedLLM excels in finetuning scenarios, showing superior running efficiency and finetuning performance. Besides, RedLLM demonstrates comparable scaling properties and context length extrapolation capability.

There are many potential future directions. We are particularly interested in exploring the scalability of RedLLM beyond 8B parameters. Our current study focused on balanced architectures, where the encoder and decoder have an equal number of layers. Investigating imbalanced architectures, such as those with a deep encoder and a shallow decoder, presents a compelling avenue. Besides, a comparative analysis of various pretraining objectives and a deeper understanding of how RedLLM extrapolates to longer sequences would be valuable.

### Acknowledgments

We’d like to thank Tianqi Liu, Tris Warkentin, and Noam Shazeer for their constructive feedback on the manuscript. We also thank the T5X team ( Roberts et al., 2023 ) for the training framework.

## Appendix A Appendix

## Instructions for reporting errors

We are continuing to improve HTML versions of papers, and your feedback helps enhance accessibility and mobile support. To report errors in the HTML that will help us improve conversion and rendering, choose any of the methods listed below:

Click the "Report Issue" ( ) button, located in the page header.

Tip: You can select the relevant text first, to include it in your report.

Our team has already identified the following issues . We appreciate your time reviewing and reporting rendering errors we may not have found yet. Your efforts will help us improve the HTML versions for all readers, because disability should not be a barrier to accessing research. Thank you for your continued support in championing open access for all.

Have a free development cycle? Help support accessibility at arXiv! Our collaborators at LaTeXML maintain a list of packages that need conversion , and welcome developer contributions .