Large Dual Encoders Are Generalizable Retrievers

arxiv:2112.07899

Original source · Paper JSON

Retrieval, embeddings, benchmarks · 2021-12-15

Cites (0) · Cited by (1)

Complete cleaned Markdown · Source Markdown including bibliography

Read in parts

1. Part 1
2. Part 2

Contents

• Large Dual Encoders Are Generalizable Retrievers
• Abstract
• 1 Introduction
• 2 Background
• 2.1 Dual Encoder and dense retrieval
• 2.2 BEIR generalization task
• 3 Generalizable T5 Retriever
• 3.1 T5 dual encoder
• 3.2 Multi-stage training
• 4 Experimental setup
• 4.1 Training Data
• Community QA.
• MS Marco.
• Natural Questions.
• 4.2 Configurations
• 4.3 Models for comparison
• 5 Evaluation Results
• 5.1 Results on MS Marco
• 5.2 Results on BEIR generalization tasks
• 5.3 Data efficiency for large retrievers
• 6 Ablation Study and Analysis
• 6.1 Effect of scaling up for different training stages
• 6.2 Importance of the fine-tuning dataset
• 6.3 Document length vs model capacity
• 7 Related Work
• Neural information retrieval.
• Dual encoders for neural retrieval.
• Zero-shot neural retrieval.
• 8 Inference latency
• 9 Conclusion
• Acknowledgments
• Appendix A More results
• A.1 Comparisons on MS Marco
• A.2 Comparison of different dual encoder pre-training strategies
• A.3 Recall on BEIR
• Instructions for reporting errors