No Train No Gain: Revisiting Efficient Training Algorithms For Transformer-based Language Models

arxiv:2307.06440

Original source · Paper JSON

Data, training, optimization · 2023-07-12

Cites (8) · Cited by (2)

Complete cleaned Markdown · Source Markdown including bibliography

Read in parts

1. Part 1
2. Part 2
3. Part 3

Contents

• No Train No Gain: Revisiting Efficient Training Algorithms For Transformer-based Language Models
• Abstract
• 1 Introduction
• 2 Comparing Efficient Training Algorithms
• The Pitfalls of Iterations
• The Pitfalls of FLOPs
• The Pitfalls of Wall-Clock Time
• Our Proposal: Reference System Time (RST)
• 2.1 The Pitfall of Comparing Intermediate Performances
• 3 Experimental Setup
• BERT: Encoder-only.
• T5: Encoder-Decoder.
• Dataset.
• Learning-rate schedule.
• Hyper-parameter search.
• Reference System.
• 4 Case Study 1: Dynamic Architectures
• 4.1 Layer stacking
• 4.2 Layer dropping
• 4.3 Results
• Training losses.
• Downstream performance.
• 5 Case Study 2: Batch Selection
• 5.1 Selective Backprop
• 5.2 RHO Loss
• 5.3 Results
• Validation loss.
• Downstream performance.
• 5.4 Ablation: Batch Selection For Free
• 6 Case Study 3: Efficient Optimizers
• Mixed Precision Training with BF16 vs. FP16
• RMS-Scaling for T5 Pre-training
• 6.1 Results
• 7 Conclusion
• Acknowledgments
• Appendix A Limitations and Future Work
• Appendix B Related Work
• B.1 Efficient Training Algorithms
• Architecture-centric strategies.
• Data-centric strategies.
• Optimization-centric strategies.
• Others.
• B.2 Efficient Training Meta-Studies
• Budget-centric recipes.
• Empirical Meta-Studies of Training Transformer-based models.
• Appendix C Hyper-Parameter Search
• C.1 Layer stacking : When To Stack
• C.2 Layer dropping : How to Drop
• C.3 Selective backprop : Selectivity Scale
• C.4 RHO loss : Mega-batch Size
• C.5 Lion : Learning Rate (LR) And Weight Decay (WD)
• C.6 Sophia : Learning Rate (LR), Weight Decay (WD) and ρ \rho
• Appendix D Additional Results
• D.1 Training BERT, Evaluating on GLUE
• D.2 Layer dropping on small datasets
• Instructions for reporting errors