Encoder-Decoder or Decoder-Only? Revisiting Encoder-Decoder Large Language Model

arxiv:2510.26622

Original source · Paper JSON

LLMs: architectures, context, training · 2025-10-30

Cites (4) · Cited by (1)

Complete cleaned Markdown · Source Markdown including bibliography

Read in parts

1. Part 1
2. Part 2

Contents

• Encoder-Decoder or Decoder-Only? Revisiting Encoder-Decoder Large Language Model
• Abstract
• 1 Introduction
• 2 Background
• 3 RedLLM: Revisiting Encoder-Decoder LLM
• 4 Setup
• Pretraining
• Finetuning
• Evaluation
• 5 Pretraining Results
• RedLLM and DecLLM scale at similar rates when increasing the compute (training Flops) and model parameters.
• RedLLM lags behind DecLLM for compute-optimal training.
• RedLLM scales slightly for few-shot learning but is poor at zero-shot; both largely underperform DecLLM.
• RedLLM shows comparable and even better length extrapolation capability than DecLLM along model scaling.
• The decoder self- and cross-attention in RedLLM show intriguing patterns under long context.
• 6 Finetuning Results
• RedLLM shows high adaptability: matching and even surpassing DecLLM across scales after finetuning.
• Bidirectional input attention improves DecLLM greatly but doesn’t change the quality-compute frontier.
• RedLLM has clear advantage over DecLLM on training and inference efficiency.
• 7 Conclusion and Future Work
• Acknowledgments
• Appendix A Appendix
• Instructions for reporting errors