T5Gemma 2: Seeing, Reading, and Understanding Longer — part 2/2

Cleaned-text code points [15561, 18033). Bibliography omitted here; source.md includes it.

We have presented T5Gemma 2, the new collection of vision-language encoder-decoder foundation model. T5Gemma 2 was built by adapting the pretrained decoder-only Gemma 3 models into encoder-decoder on ∼ \sim 2T UL2 tokens. We ablated several architecture designs and integrated two proposal to save model parameters: tied embeddings across encoder and decoder, and merged attention unifying decoder self- and cross-attention sub-layers. The resulting decoder architecture resembles the encoder architecture, facilitating the adaptation from decoder-only models.

T5Gemma 2 accepts text and/or image as inputs to the encoder, and generates response text from the decoder. We evaluated the models across a range of benchmarks, covering five capabilities: reasoning and factuality, stem and coding, multilingual, multimodal and long-context. In general, T5Gemma 2 shows competitive pretraining performance than Gemma 3 and improved post-training performance across capabilities.

Especially, T5Gemma 2 shows strong multimodal and long-context performance, thanks to its encoder-decoder architecture. Unlike decoder-only LLMs, T5Gemma 2 has a dedicated set of encoder parameters for input/vision or prompt understanding. Its cross-attention over the encoder outputs also endows it with better ability on retrieving relevant information from the inputs.

Beyond standalone usage, T5Gemma 2 serves as a robust foundation for training high-quality downstream embedding models. For example, EmbeddingGemma ( Vera et al., 2025 ) leverages T5Gemma 2 checkpoints to achieve state-of-the-art performance on text retrieval benchmarks.

We released all three pretrained checkpoints (270M-270M, 1B-1B and 4B-4B) to facilitate the evaluation, adaptation and research by the community. Note, to the best of our knowledge, T5Gemma 2 presents itself as the first capable long-context encoder-decoder LLMs (up to 128K) in the community. By offering novel insights into encoder-decoder LLMs, we hope this work to be a catalyst for future innovation, ultimately benefiting the development of more sophisticated and powerful LLMs.

## Acknowledgments

We’d like to thank Lechao Xiao for his insightful comments. Our work is made possible by the dedication and efforts of numerous teams at Google. We would like to acknowledge the support from the following teams: DevX, Gemini Infrastructure, Gemini Safety, Gemma, Google Cloud, Google Research Responsible AI, Kaggle, and Gemini Encoder-heavy.