Knowledge Mechanisms in Large Language Models: A Survey and Perspective — part 5/6

Cleaned-text code points [60498, 75725). Bibliography omitted here; source.md includes it.

Remarks : LLMs may improve their architecture and mechanisms for knowledge learning, storage, and expression, drawing inspiration from neuroscience. Besides, cognitive science and psychology provide promising alternatives for sophisticated intelligence, emergent capabilities and behaviors in evolution. Educational studies can inspire the learning strategy of LLMs, navigating conflicts and integrating knowledge during their evolution.

## 7 Future Directions

### 7.1 Parametric VS. Non-Parametric Knowledge

LLMs can be conceptualized as parametric knowledge stores, where the parameters of the model—typically the weights of the neural network—encode a representation of the world’s knowledge. This parametric approach to knowledge storage means that the knowledge is implicitly embedded within the model’s architecture, and it can be retrieved and manipulated through the computational processes of the neural network Allen-Zhu and Li (2023b) . In contrast, non-parametric knowledge storage involves methods where the knowledge is explicitly represented and can be directly accessed. Examples of non-parametric knowledge storage include knowledge graphs, databases, and symbolic reasoning systems, where knowledge is represented as discrete symbols or facts. Parametric knowledge enables LLMs to deeply compress and integrate information Huang et al. (2024d) ; Shwartz-Ziv and LeCun (2024) , allowing them to generalize and apply this knowledge across various contexts. This is akin to LLMs mastering the mathematical operation rule of “mod” through parametric knowledge, enabling them to generalize and seamlessly solve all mod-related problems Pearce et al. (2023) ; Hu et al. (2024) . Conversely, non-parametric knowledge requires extensive searches across the knowledge space for each user query. Subsequently, Wang et al. (2024a) also prove that non-parametric knowledge severely fails in complex reasoning tasks, with accuracy levels approaching random guessing. Unfortunately, parametric knowledge within LLMs is opaque, often encountering challenges such as interpretability issues, outdated information, hallucinations, and security concerns.

Addressing these issues often requires leveraging external non-parametric knowledge, which offers transparency, flexibility, adaptability, and ease of operation. However, augmenting parametric knowledge in LLMs with non-parametric knowledge Yang et al. (2024b) ; Luo et al. (2023) ; Wen et al. (2023) ; Ko et al. (2024) remains an ongoing challenge due to retrieval accuracy from haystack, context lengths, and resources 9 9 9 On the one hand, storing large amounts of non-parametric knowledge requires a lot of space and high maintenance costs. On the other hand, retrieving information for each user query is very resource-intensive. limitations Shang et al. (2024) ; Zhao et al. (2024b) . Besides, simultaneously retrieving relevant information from a long context and conducting reasoning is nearly impossible in reasoning-in-a-haystack experiments Shang et al. (2024) . Similarly, augmenting non-parametric knowledge —either by distilling knowledge from an LLM’s parametric knowledge West et al. (2022) ; Kazemi et al. (2023) or by using it to parse text directly Zhang et al. (2023b) —also poses significant challenges. Moreover, Yang et al. (2024b) propose a novel explicit memory that lies between parametric and non-parametric knowledge . LLM with explicit memory enjoys a smaller parameter size and lower resource consumption for retrieving external non-parametric knowledge.

Generally, inspired by the knowledge mechanisms analysis in LLMs, we have the potential to develop more architectural and learning strategies for organizing knowledge within LLMs . These efficient LLMs Sastry et al. (2024) are advancing toward lower GPU, computation, and storage resource requirements, as well as smaller model sizes by combining the strengths of parametric and non-parametric knowledge Yang et al. (2024b) ; Momeni et al. (2024) ; Chen (2024) ; Pan et al. (2024b) ; Pan et al. (2023b) .

### 7.2 Embodied Intelligence

The current LLM still cannot be regarded as a truly intelligent creature Bender and Koller (2020) ; Bisk et al. (2020) . The process of human language acquisition is not merely a passive process of listening to language. Instead, it is an active and interactive process that involves engagement with the physical world and communication with other people. To enhance the current LLM’s capabilities and transform it into a powerful agent, it is necessary to enable it to learn from multimodal information and interact with the environment and humans.

##### Multimodal LLMs.

The integration of multiple modalities is a critical challenge in the field of LLMs and embodied AI. While LLMs have demonstrated impressive capabilities when processing language data, their ability to seamlessly incorporate and synthesize information from other modalities such as images, speech, and video is still an area of active research. However, the current multi-modal model faces challenges, particularly in complex reasoning tasks that require understanding and integrating information from both text and images.

Recent studies Huang et al. (2024a) ; Chen et al. (2024c) have highlighted the discrepancy between the model’s performance in language tasks and its ability to integrate knowledge from different modalities effectively. These findings suggest that current models often prioritize linguistic information, failing to fully exploit the synergistic potential of multimodal data Wang et al. (2024c) . There are some pioneering efforts in this direction Pan et al. (2024a) ; Schwettmann et al. (2023a) , aiming to uncover the mechanisms by which multi-modal models store and retrieve information. Despite these advancements, there is still a need for further exploration to deepen our understanding of multi-modal knowledge storage.

##### Self-evolution.

As discussed in the previous part, current language models are mainly based on tuning to gain knowledge, which requires a lot of training and high-quality data. These learnings are passive whereas, to be a human, evolution usually also undergoes communication and interaction. As an intelligent agent, the models should be able to learn through interactions and learn by themselves spontaneously. Recently, some work has attempted to enable the model to learn by themselves Zhang et al. (2024b) or learn by interaction with the environment Xu et al. (2024a) ; Xi et al. (2024) . By integrating self-evolving mechanisms, models can continuously update their knowledge base and improve their understanding without relying solely on manually curated datasets. This not only reduces the dependency on large-scale labeled data but also allows the models to adapt to evolving linguistic norms and cultural contexts over time.

### 7.3 Domain LLMs

The success of general-purpose LLMs has indeed inspired the development of domain-specific models that are tailored to particular areas of knowledge Calderon and Reichart (2024) , such as biomedicine Yu et al. (2024a) ; Moutakanni et al. (2024) , finance Yang et al. (2023) , geoscience Deng et al. (2023) , ocean science Bi et al. (2024) , etc. However, unlike human language, the knowledge of these different domains bears specific characteristics. It remains unclear whether LLMs can acquire complex scientific knowledge or if such knowledge still resides within the realm of current dark knowledge. Furthermore, does domain-specific knowledge such as mathematics share the same underlying mechanisms as textual knowledge Bengio and Malkin (2024) , or does it exhibit more intricate mechanisms of knowledge acquisition? Currently, there is a relative lack of research focusing on the mechanism of these domain-specific knowledge and there is an increasing recognition of the importance of developing a deeper understanding of these mechanisms.

Data sparsity and diversity in domain-specific models pose another challenge. Sparsity is usually caused by confidentiality, privacy, and the cost of acquisition in specialized fields. As for diversity, the presentation of knowledge varies across different fields. For instance, in the biomedical domain, knowledge includes complex biological concepts such as the structure and function of proteins and molecules. This requires models to integrate understanding that extends beyond natural language, often involving graphical representations like chemical structures, which cannot be directly expressed in text. Similarly, in fields such as finance and law Lai et al. (2023) , models must engage in sophisticated reasoning and decision-making processes based on domain-specific knowledge. Hence, the critical tasks of collecting high-quality data for domain-specific models (including synthetic data generation) and effectively embedding domain knowledge into LLMs require immediate attention.

## 8 Conclusion

In this paper, we propose a novel knowledge mechanism analysis taxonomy and review knowledge evolution. We further discuss knowledge utilization issues, as well as unexplored dark knowledge. We hope these insights may inspire some promising directions for future research and shed light on more powerful and trustworthy models.

## Limitations

This work has some limitations as follows:

##### Hypothesis

Despite reviewing a large body of literature and proposing several promising hypotheses, there are still some limitations. On the one hand, there may be other hypotheses for knowledge utilization and evolution in LLMs. On the other hand, the accuracy of these hypotheses requires further exploration and validation over time.

##### Knowledge

There are various forms of knowledge representation. However, due to current research constraints, this paper does not delve into space Li et al. (2024f) , time Gurnee and Tegmark (2023) , event-based knowledge, and geoscience Lin et al. (2024) .

##### Reference

The field of knowledge mechanisms is developing rapidly and this paper may miss some important references. Additionally, due to the page limit, we have omit certain technical details. We will continue to pay attention to and supplement new works.

##### Models

Despite mentioning artificial neural models in this paper, knowledge mechanism analysis focuses on LLMs. We will continue to pay attention to other modal models progresses. Besides, all existing work has not considered models larger than 100 billion parameters. Whether the knowledge mechanisms within large-scale models are consistent with smaller ones remains to be studied.

## Ethics Statement

We anticipate no ethical or societal implications arising from our research. However, we acknowledge that the internal mechanisms of large language models might be exploited for malicious purposes. We believe such malicious applications can be prevented through model access and legislative regulation. More critically, a transparent model contributes to the development of safer and more reliable general artificial intelligence.

## Acknowledgements

We would like to express gratitude to the anonymous reviewers for their kind comments. This work was supported by the National Natural Science Foundation of China (No. 62206246, No. NSFCU23B2055, No. NSFCU19B2027), the Fundamental Research Funds for the Central Universities (226-2023-00138), Zhejiang Provincial Natural Science Foundation of China (No. LGG22F030011), Yongjiang Talent Introduction Programme (2021A-156-G), Information Technology Center and State Key Lab of CAD&CG, Zhejiang University, and NUS-NCS Joint Laboratory (A-0008542-00-00).

## Appendix A Comparison of Methods for Knowledge Evolution

Note that due to the page limit, § 4 does not provide a detailed enumeration of various techniques and details, such as machine unlearning and knowledge augmentation. Hence, we briefly outline common methods during post-training stage in this section and illustrate their associations and differences Zhang et al. (2024a) in Fig 5 .

• Continual Learning aims to continually acquire new skills and learn new tasks while retaining previously acquired knowledge.

• Parameter-efficient Fine-tuning (PET) Zhang et al. (2019) only updates a minimal set of parameters instead of full fine-tuning. A promising strategy is LoRA Hu et al. (2022) .

• Knowledge Augmentation is proposed to assist the model in handling unknown knowledge for LLMs Zhang et al. (2019) ; Han et al. (2022) . RAG Huang and Huang (2024) is the is the most prevalent methods. Beside, knowledge augmentation also includes prompt engineering Gu et al. (2023) ; Kraljic and Lahav (2024) ; Liang et al. (2023b) and in-context learning Luo et al. (2024a) .

• Machine Unlearning Nguyen et al. (2022) ; Tian et al. (2024) ; Liu et al. (2024d) focuses on discarding undesirable behaviors from LLMs.

• Editing , including knowledge editing Zhang et al. (2024a) and representation editing Wu et al. (2024) , aims to enable quick and precise modifications to the LLMs. Usually, editing first identifies the knowledge location in LLMs and then precisely modifies model behavior through a few instances.

## Appendix B Universality Intelligence

To validate the hypotheses in this paper across different architectures, we first introduce other popular model architectures in § B.1 , and observe the generalizability of our hypotheses across other model architectures in § B.2 . Besides, recent work further claims that models trained with different data, modalities, and objectives are converging to shared representation spaces Huh et al. (2024) . Artificial and biological neural networks 10 10 10 Unless otherwise specified as biological networks, the terms models, neural networks, neural models, and machines refer to artificial neural networks. also share similar features and circuits, suggesting a universal underlying mechanism Sucholutsky et al. (2023) ; Chan et al. (2023) ; Kornblith et al. (2019) . Therefore, analogous to biological taxonomy, we introduce artificial neural model family and discuss the potential universality intelligence in the future in § B.3 .

### B.1 Model Architecture

#### B.1.1 Transformer

##### MLP

The Multilayer Perceptron (MLP) is a crucial component in neural networks, usually comprising multiple fully connected layers. Within the Transformer architecture, the MLP plays a vital role in applying nonlinear transformations to the input hidden states, thereby enriching the model’s capacity for expression. More precisely, every MLP block involves two linear transformations separated by a point-wise activation function σ \sigma : MLP l ​ ( h l ) = σ ⁡ ( W K l ​ h l ) ​ W V l , \displaystyle\text{MLP}^{l}\left(h^{l}\right)=\sigma\left(W_{K}^{l}h^{l}\right)W_{V}^{l}, (6) where σ \sigma is the point-wise activation function, typically a non-linear function such as ReLU or GELU. W K l W_{K}^{l} is the weight matrix for the first linear transformation in the l l -th layer, mapping the input hidden state h l h^{l} to an intermediate representation. W V l W_{V}^{l} is the weight matrix for the second linear transformation in the l l -th layer, transforming the intermediate representation to the output of the MLP block.

##### Attention