T5Gemma 2: Seeing, Reading, and Understanding Longer

arxiv:2512.14856

Original source · Paper JSON

LLMs: architectures, context, training · 2025-12-16

Cites (2) · Cited by (0)

Complete cleaned Markdown · Source Markdown including bibliography

Read in parts

1. Part 1
2. Part 2

Contents

• T5Gemma 2: Seeing, Reading, and Understanding Longer
• Abstract
• 1 Introduction
• 2 T5Gemma 2
• Tied Embedding
• Merged Attention
• Rejected Ablation : Cross-Attention on Global Layers Only
• 3 Setup
• 3.1 Pretraining
• Data
• Optimization
• 3.2 Post-training
• 3.3 Evaluation
• 4 Results
• PrefixLM+KD, UL2, vs. UL2+KD: Do distillation and training objective matter?
• Text-only LLMs can be adapted into strong multimodal and long-context encoder-decoder models.
• T5Gemma 2 achieves competitive pretraining performance and improved post-training performance than Gemma 3.
• 5 Conclusion
• Acknowledgments