GLU Variants Improve Transformer

arxiv:2002.05202

Original source · Paper JSON

LLMs: architectures, context, training · 2020-02-12

Cites (1) · Cited by (16)

Complete cleaned Markdown · Source Markdown including bibliography

Read in parts

1. Part 1

Contents

• GLU Variants Improve Transformer
• Abstract
• 1 Introduction
• 2 Gated Linear Units (GLU) and Variants
• 3 Experiments on Text-to-Text Transfer Transformer (T5)
• 3.1 Model Architecture
• 3.2 Pre-Training and Perplexity Results
• 3.3 Fine-Tuning
• 4 Conclusions