Knowledge Mechanisms in Large Language Models: A Survey and Perspective

arxiv:2407.15017

Original source · Paper JSON

Reasoning and the "physics" of language models · 2024-07-22

Cites (14) · Cited by (0)

Complete cleaned Markdown · Source Markdown including bibliography

Read in parts

1. Part 1
2. Part 2
3. Part 3
4. Part 4
5. Part 5
6. Part 6

Contents

• Knowledge Mechanisms in Large Language Models: A Survey and Perspective
• Abstract
• 1 Introduction
• Comparison with Existing Surveys
• 2 Preliminary
• 2.1 Knowledge Scope
• 2.2 Definition of Knowledge in LLMs
• 2.3 The Architecture of LLMs
• Transformer-based architecture.
• Other architectures.
• 2.4 Knowledge Analysis Methods
• Observation-based methods.
• Intervention-based methods.
• 3 Knowledge Utilization in LLMs
• 3.1 Memorization
• 3.2 Comprehension and Application
• 3.3 Creation
• 3.4 Comparison of Different Mechanism Analysis Methods
• 4 Knowledge Evolution in LLMs
• 4.1 Individual Evolution
• 4.2 Group Evolution
• 4.3 Comparison of Different Evolution Strategies
• 5 Application of Knowledge Mechanism
• 5.1 Efficient LLMs
• 5.2 Trustworthy LLMs
• 6 Discussion
• 6.1 What Knowledge Have LLMs Learned?
• 6.2 Why Is Learned Knowledge Fragile?
• 6.3 Does Difficult-to-Learn “Dark Knowledge” Exist?
• 6.4 How to Explore More Knowledge from Interdisciplinary Inspiration?
• 7 Future Directions
• 7.1 Parametric VS. Non-Parametric Knowledge
• 7.2 Embodied Intelligence
• Multimodal LLMs.
• Self-evolution.
• 7.3 Domain LLMs
• 8 Conclusion
• Limitations
• Hypothesis
• Knowledge
• Reference
• Models
• Ethics Statement
• Acknowledgements
• Appendix A Comparison of Methods for Knowledge Evolution
• Appendix B Universality Intelligence
• B.1 Model Architecture
• B.1.1 Transformer
• MLP
• Attention
• Variants of Transformer
• B.1.2 SSM
• Mamba
• B.1.3 Vision and Multi-modal Models
• B.2 Knowledge Mechanisms in Other Architectures
• B.3 Machine and Human
• Appendix C Tools for Mechanism Analysis
• Instructions for reporting errors