Physics of Language Models: Part 3.1, Knowledge Storage and Extraction

arxiv:2309.14316

Original source · Paper JSON

Reasoning and the "physics" of language models · 2023-09-25

Cites (4) · Cited by (13)

Complete cleaned Markdown · Source Markdown including bibliography

Read in parts

1. Part 1
2. Part 2
3. Part 3
4. Part 4
5. Part 5
6. Part 6

Contents

• Physics of Language Models: Part 3.1, Knowledge Storage and Extraction Thanks: Project page: https://physics.allen-zhu.com/part-3-knowledge/part-3-1 . An extended video of this paper is available at https://youtu.be/YSHzKmEianc . V1 was circulated internally at Meta on Sep 18, 2023, and appeared on arXiv on Sep 25, 2023. V2 is nearly identical to V1, with minor corrections to author names and writing. V3 includes additional Llama experiments and further writing improvements. We would like to thank Lin Xiao, Chunting Zhou, Tianyi Peng, Xiaodong Liu, and Zhijie Zhou for many helpful conversations. We would like to extend special thanks to Nabib Ahmed, Giri Anantharaman, Lucca Bertoncini, Henry Estela, Liao Hu, Caleb Ho, Wil Johnson, Apostolos Kokolis, and Shubho Sengupta from Meta FAIR, as well as Ian Clark, Gourab De, Anmol Mann, and Max Pfeifer from W&B ; without their invaluable support, the experiments in this paper would not have been possible.
• Abstract
• 1 Introduction
• 1.1 Related Work
• 2 Result 0: Our Dataset Families
• 2.1 Training Details
• 3 Result 1: Mixed Training Enables Knowledge Extraction
• 3.1 Model’s Abnormal Learning Behavior
• Remark 3.1 .
• 4 Result 2-3: BIO Pretrain + QA Instruction Finetune
• 4.1 Result 2: Model Fails to Extract Knowledge After BIO Pretrain
• 4.2 Result 3: Knowledge Augmentation
• Remark .
• 5 Results 4-5: Knowledge Probes on the BIO Pretrained Model
• 5.1 Result 4: Position-Based Probing
• 5.1.1 Closer P-Probing at Knowledge Dependency
• Remark 5.1 .
• 5.1.2 P-Probing Extensions
• 5.2 Result 5: Query-Based Probing
• 6 Result 6: Celebrity Can Help Minority
• Remark 6.1 .
• 7 Result 7: Knowledge Storage for Bidirectional Models
• 8 Conclusion
• Appendix A Details on Data Preparation
• A.1 BIO dataset bioS
• A.1.1 bioS couple
• A.2 BIO dataset bioR
• Appendix B Details on Model Architecture
• Appendix C Details on Pretrain and Mixed Training
• Remark C.1 .
• Appendix D Details on QA Finetune
• Appendix E Details on P-probing
• E.1 Details on Closer P-Probing
• Appendix F Details on Q-probing
• Appendix G Details on Celebrity Augementation
• Appendix H Details on BERT Experiment