Cramming: Training a Language Model on a Single GPU in One Day

arxiv:2212.14034

Original source · Paper JSON

Data, training, optimization · 2022-12-28

Cites (3) · Cited by (3)

Complete cleaned Markdown · Source Markdown including bibliography

Read in parts

1. Part 1
2. Part 2
3. Part 3

Contents

• Cramming: Training a Language Model on a Single GPU in One Day
• Abstract
• 1 Scaling Up and Scaling Down
• 2 Tying our hands behind our back: A setup with limited compute
• 3 Related Work on Efficient Transformers
• How long does it take to train BERT?
• Studies of Efficient Transformers
• Scaling Laws
• 4 Investigations
• 4.1 Implementation Details
• Initial Data Setup
• 4.2 Modifying the Architecture
• Scaling laws hold in the low-resource regime
• Exploiting the scaling law.
• Attention Block:
• Feedforward Block:
• Embedding:
• Layer Structure:
• Head Block:
• 4.3 Modifying the Training Setup
• Objective:
• Choice of Optimizer:
• Learning Rate Schedule and Peak:
• Batch Size Schedule:
• Dropping Dropout
• 4.4 Optimizing the Dataset
• Vocabulary Size
• 5 Finetuning Performance on GLUE
• 5.1 Ablation - Which Changes Really Mattered?
• 5.2 What Happens When Training Longer?
• 6 Limitations
• 7 Conclusions
• Reproducibility Statement
• Appendix
• Appendix A Other Modifications
• Appendix B Additional Information
• B.1 References for Table 1
• Instructions for reporting errors