Encoder-Decoder Gemma: Improving the Quality-Efficiency Trade-Off via Adaptation

arxiv:2504.06225

Original source · Paper JSON

LLMs: architectures, context, training · 2025-04-08

Cites (1) · Cited by (1)

Complete cleaned Markdown · Source Markdown including bibliography

Read in parts

1. Part 1
2. Part 2

Contents

• Encoder-Decoder Gemma: Improving the Quality-Efficiency Trade-Off via Adaptation
• Abstract
• 1 Introduction
• 2 Related Work
• 3 Approach: Encoder-Decoder Adaptation
• 3.1 Architecture
• 3.2 Initialization
• 3.3 Pretraining Objective
• 4 Setup
• Data Setting
• Model Setting
• Evaluation
• 5 Results
• The encoder-decoder adaptation converges rapidly, particularly for balanced architectures.
• Pretraining objective matters: UL2 and PrefixLM show different characteristics.
• Encoder-decoder LLMs outperform decoder-only LLMs especially after instruction tuning.
• Encoder-decoder LLMs balance quality and inference efficiency more effectively.
• 6 Discussion
• Is the improvement after the adaptation simply due to the extra pretraining compute?
• Does cross-attention warmup matter for unbalanced encoder-decoder?
• Can we switch from grouped-query attention to multi-head self attention for the encoder?
• Does bidirectional self-attention matter for the encoder?
• Would pretraining encoder-decoder LLMs from scratch yield better performance?
• Is IT/SuperGLUE score predicable from PT score?
• Can we get the best of both worlds from PrefixLM and UL2?
• 7 Conclusion and Future Work
• Acknowledgements