Encoder-Decoder or Decoder-Only? Revisiting Encoder-Decoder Large Language Model — part 2/2

Cleaned-text code points [15271, 28925). Bibliography omitted here; source.md includes it.

We also evaluate zero/few-shot performance of pretrained and finetuned models. We follow Longpre et al. (2023) and extend the evaluation to 13 tasks: BoolQ ( Clark et al., 2019 ) , HellaSwag ( Zellers et al., 2019 ) , RTE ( Bentivogli et al., 2009 ) , ARC easy and challenge ( Clark et al., 2018 ) , ANLI R1, R2 and R3 ( Nie et al., 2020 ) , CommonsenseQA ( Talmor et al., 2019 ) , StrategyQA ( Geva et al., 2021 ) , TriviaQA ( Joshi et al., 2017 ) , UnifiedQA ( Khashabi et al., 2020 ) , MMLU ( Hendrycks et al., 2021 , CoT) , GSM8K ( Cobbe et al., 2021 ) , BIG-Bench Hard ( bench authors, 2023 , BBH CoT,) , and WMT ( Kocmi et al., 2022 , English ↔ \leftrightarrow German, English ↔ \leftrightarrow Chinese,) . We report accuracy except WMT for which we use ChrF. We employ 5 shots for few-shot evaluation except BBH (3 shots), GSM8K (8 shots), CommonsenseQA (7 shots), and StrategyQA (6 shots). We adopt greedy decoding for all models. We calculate averaged score over all tasks for model comparison, and report the best results for finetuning.

## 5 Pretraining Results

#### RedLLM and DecLLM scale at similar rates when increasing the compute (training Flops) and model parameters.

The substantial differences in architecture and pretraining objective make it challenging to find a common ground comparing RedLLM and DecLLM. Doubling model parameters for RedLLM as in ( Wang et al., 2022 ; Tay et al., 2022 ) is arguably unfair. We instead resort to scaling law ( Kaplan et al., 2020 ) and compare them under two factors: model parameters and training Flops. Figure 2 and 10 show the results; Table 2(b) summarizes the scaling exponents. We also show the pretraining loss curve in Figure 11 .

Interestingly, RedLLM shows comparable scaling capability to DecLLM regardless of evaluation domains, as demonstrated by similar scaling exponents. DecLLM is more parameter efficient, outperforming RedLLM by a consistent gap under the same amount of parameters. However, this comes at the cost of computational inefficiency, as it requires approximately double the Flops to train under similar conditions. When shifting the comparison basis to compute, the quality gap almost disappears where both scaling curves nearly overlap.

#### RedLLM lags behind DecLLM for compute-optimal training.

We next investigate how effectively different models utilize the compute, as measured by PPL during the pretraining. Intuitively, a model achieving lower PPL under the same compute budget is preferred. Figure 3 and 13 show the comparison between RedLLM and DecLLM with different compute budgets. We first plot the PPL-to-compute curve for all models, where the bottom left region indicates the Pareto frontier. We then use this information to fit a compute-optimal scaling law function ( Hoffmann et al., 2022 ) for both RedLLM and DecLLM, and plot the isoFLOP for both. Figure 3 (and 13 ) shows that while RedLLM shows a slight edge in low-compute settings, DecLLM’s advantage becomes dominant as the compute scales up. This superior scaling behavior might come from its causal LM objective, which endows DecLLM with higher efficiency in utilizing training tokens. Still, the quality gap between RedLLM and DecLLM is not that substantial and narrows with the increase of compute.

#### RedLLM scales slightly for few-shot learning but is poor at zero-shot; both largely underperform DecLLM.

While PPL has been widely used for measuring LLM capability, it may not be indicative enough for the downstream task-solving ability ( Kuribayashi et al., 2021 ; Hu et al., 2024 ) . Apart from PPL, we further examine models’ zero/few-shot performance as shown in Figure 4 . RedLLM and DecLLM both achieve non-trivial few-shot performance, demonstrating their capability of in-context learning. DecLLM yields consistently better performance as scaling up model and data size, for both zero-shot and few-shot learning. In contrast, RedLLM performs badly at zero-shot learning, and only shows slight benefits from the scaling for few-shot learning. Its inferior zero-shot results, nevertheless, may not result from RedLLM’s insufficient capability but from an inadequate way of zero-shot prompting ( Patel et al., 2023 ) . In general, RedLLM falls behind DecLLM substantially at zero/few-shot learning during pretraining, resonating with the findings of Wang et al. (2022) . This still holds true even when their PPL scores are comparable, e.g., RedLLM 8B vs. DecLLM 4B.

#### RedLLM shows comparable and even better length extrapolation capability than DecLLM along model scaling.

One strength of DecLLM with rotary embedding is its length extrapolation ability, i.e., being able to handle sequences beyond pretraining context length. This raises the question of how RedLLM performs on long sequence modeling and we show the results in Figure 5 and 12 . In general, PPL goes up – becomes worse – for both RedLLM and DecLLM as context length increases, regardless of prefix lengths and domains.

DecLLM maintains relatively stable PPL up to a context length of 4096 tokens, i.e., twice the pretraining length. Beyond this point, PPL starts increasing, eventually exhibiting divergent behavior. We note that the larger DecLLM is, the more rapid PPL escalates as sequence length extends. In contrast, RedLLM shows a surprisingly smoother increase in PPL, regardless of model sizes, demonstrating a stronger ability to extrapolate to longer sequences. Nevertheless, it often underperforms DecLLM on sequences shorter than 4K tokens.

Interestingly, the impact of prefix length is marginal on the extrapolation behavior except RedLLM at prefix length of 1. RedLLM suffers from significant quality drop on short sequences when reducing the prefix length to 1 since the capacity from its encoder becomes limited. As model scales up, however, the gap gets largely narrowed, perhaps due to the improved decoder capacity.

#### The decoder self- and cross-attention in RedLLM show intriguing patterns under long context.

The above results motivate us to dive deeper into the models for a better understanding of what happened under long context. We find from Figure 6 (top) that the groundtruth target token’s log probability often decreases as token position increases. In contrast to RedLLM, DecLLM tends to suffer from a sharp probability drop after some point. We further inspect the decoder self-attention with the suspect that self-attention weights might not be functioning as intended. Figure 7(b) and 7(c) show that the decoder self-attention in RedLLM and DecLLM follows a clear “ locality ” structure where tokens favor attending to itself and nearby tokens, but the strength of locality weakens with the increase of token position particularly in DecLLM– a phenomenon we called locality decay . To verify this finding, we calculated the local attention probability, i.e., summed attention weights over a local window size of 5. Figure 6 (bottom) shows that the decay behavior occurs in both RedLLM and DecLLM, while DecLLM suffers more.

Figure 7(a) shows that the pattern of cross-attention weights differs significantly from that of self-attention weights. There is always a subset of input tokens attended by target tokens in RedLLM, and this subset varies little ( Zhang et al., 2020 ) . Such diverse attention may help RedLLM to capture some distant information under long context modeling, but may also increase learning difficulty as the input-target dependency in language modeling is of higher uncertainty. Still, the underlying reason behind RedLLM’s long context behavior remains unclear, which we leave to the future.

## 6 Finetuning Results

#### RedLLM shows high adaptability: matching and even surpassing DecLLM across scales after finetuning.

While RedLLM’s zero/few-shot performance lags behind DecLLM in pretraining, finetuning changes the situation. Figure 8 shows the averaged scaling results for finetuning along with pretraining under model parameters and inference Flops, and Table 3 lists detailed numbers. RedLLM catches up with DecLLM after finetuning, even under the same amount of model parameters (where RedLLM produces higher PPL than DecLLM). When switching the comparison basis to the inference Flops, RedLLM shows significant advantage, nearly dominating the quality-compute Pareto frontier. The performance gap between zero-shot and few-shot evaluation is also largely reduced. Apart from averaged results, we add per-task performance in Figures 14 , 15 and 16 . The results vary across tasks, e.g., RedLLM largely surpasses DecLLM on ANLI, while DecLLM excels at ARC. These differences should be carefully considered when comparing both models.

Note Wang et al. (2022) also observed similar results. However, their experiments were based on a 11B encoder-decoder model and 4.8B decoder-only model. In contrast, we present the scaling landscape across multiple model sizes. We further demonstrate the feasibility of achieving comparable performance to DecLLM under similar model sizes.

In addition, these results suggest that the adaptability of different models varies greatly. Models with worse pretraining performance may stand out after finetuning, such as RedLLM. We conjecture that the superior pretraining performance of DecLLM is mostly caused by the higher degree of matching between its pretraining objective and the downstream evaluation, rather than its stronger capability. Surprisingly, PPL also fails to capture a model’s adaptability, echoing with Hu et al. (2024) .

#### Bidirectional input attention improves DecLLM greatly but doesn’t change the quality-compute frontier.

An important difference between RedLLM and DecLLM at finetuning is the bidirectional input attention (BiAttn) in RedLLM’s encoder, which allows input tokens attending to each other. Such a structure often improves input understanding, and has proven useful in various settings including DecLLM ( Wang et al., 2022 ; Zhang et al., 2022a ) . This may explain why DecLLM underperforms RedLLM after finetuning.

We thus enable this structure for DecLLM. Figure 8 and Table 3 show that the performance of DecLLM gets substantially improved. Interestingly, while DecLLM + BiAttn largely outperforms RedLLM at zero-shot learning, their few-shot quality gap becomes smaller to negligible. Still, the overall quality gap among different models is much smaller compared to the pretraining performance gap, and the leading position of RedLLM with respect to the quality-compute trade-off remains.

#### RedLLM has clear advantage over DecLLM on training and inference efficiency.

Beyond theoretical Flops analysis, our empirical efficiency analysis (Figure 9 ) reveals a striking advantage for RedLLM in both training and inference throughput. This advantage becomes even more pronounced as model size increases, particularly during pretraining, where RedLLM significantly outpaces DecLLM. This remarkable efficiency, together with RedLLM’s promising finetuning performance, underscores its significance, particularly on the fact that instruction tuning and alignment have become an essential part of LLM development ( Ouyang et al., 2022 ) .

## 7 Conclusion and Future Work

Encoder-decoder or decoder-only for LLM? We answered this question from the scaling perspective and provided a comprehensive analysis over multiple dimensions. We conducted large-scale pretraining and finetuning experiments using RedPajama V1 and FLAN, with models ranging from 150M to 8B parameters. Our results show distinct strengths for each architecture. DecLLM presents unique strengths during pretraining, including compute-optimal scaling, zero/few-shot learning, and strong training stability. By contrast, RedLLM excels in finetuning scenarios, showing superior running efficiency and finetuning performance. Besides, RedLLM demonstrates comparable scaling properties and context length extrapolation capability.

There are many potential future directions. We are particularly interested in exploring the scalability of RedLLM beyond 8B parameters. Our current study focused on balanced architectures, where the encoder and decoder have an equal number of layers. Investigating imbalanced architectures, such as those with a deep encoder and a shallow decoder, presents a compelling avenue. Besides, a comparative analysis of various pretraining objectives and a deeper understanding of how RedLLM extrapolates to longer sequences would be valuable.

### Acknowledgments

We’d like to thank Tianqi Liu, Tris Warkentin, and Noam Shazeer for their constructive feedback on the manuscript. We also thank the T5X team ( Roberts et al., 2023 ) for the training framework.

## Appendix A Appendix

## Instructions for reporting errors

We are continuing to improve HTML versions of papers, and your feedback helps enhance accessibility and mobile support. To report errors in the HTML that will help us improve conversion and rendering, choose any of the methods listed below:

Click the "Report Issue" ( ) button, located in the page header.

Tip: You can select the relevant text first, to include it in your report.

Our team has already identified the following issues . We appreciate your time reviewing and reporting rendering errors we may not have found yet. Your efforts will help us improve the HTML versions for all readers, because disability should not be a barrier to accessing research. Thank you for your continued support in championing open access for all.

Have a free development cycle? Help support accessibility at arXiv! Our collaborators at LaTeXML maintain a list of packages that need conversion , and welcome developer contributions .