Smarter, Better, Faster, Longer: A Modern Bidirectional Encoder for Fast, Memory Efficient, and Long Context Finetuning and Inference

arxiv:2412.13663

Original source · Paper JSON

LLMs: architectures, context, training · 2024-12-18

Cites (4) · Cited by (0)

Complete cleaned Markdown · Source Markdown including bibliography

Read in parts

1. Part 1
2. Part 2
3. Part 3
4. Part 4

Contents

• Smarter, Better, Faster, Longer : A Modern Bidirectional Encoder for Fast, Memory Efficient, and Long Context Finetuning and Inference
• Abstract
• 1 Introduction
• 2 Methods
• 2.1 Architectural Improvements
• 2.1.1 Modern Transformer
• 2.1.2 Efficiency Improvements
• 2.1.3 Model Design
• 2.2 Training
• 2.2.1 Data
• 2.2.2 Training Settings
• 3 Downstream Evaluation
• 3.1 Evaluation Setting
• 3.1.1 Natural Language Understanding
• 3.1.2 Text Retrieval
• 3.1.3 Long-Context Text Retrieval
• 3.1.4 Code Retrieval
• 3.2 Downstream Results and Discussion
• 4 Efficiency
• 4.1 Evaluation Setting
• 4.2 Results
• 5 Conclusion
• 6 Limitations
• 7 Acknowledgements
• 8 Contribution Statement
• Appendix A Training Settings
• A.1 Batch Size Schedule
• A.2 Weight Tiling
• A.3 Weight Decay
• A.4 Final Checkpoints
• Appendix B Model Design
• Appendix C Training Log
• C.1 Sampling Issue
• C.2 Large Rollback
• Appendix D Architecture ablations
• Appendix E Extended results
• E.1 Full GLUE results
• E.2 Full BEIR results
• Appendix F Efficiency
• Appendix G Licensing
• Instructions for reporting errors