Knowledge Mechanisms in Large Language Models: A Survey and Perspective — part 1/6

Cleaned-text code points [0, 14825). Bibliography omitted here; source.md includes it.

# Knowledge Mechanisms in Large Language Models: A Survey and Perspective

###### Abstract

Understanding knowledge mechanisms in Large Language Models (LLMs) is crucial for advancing towards trustworthy AGI. This paper reviews knowledge mechanism analysis from a novel taxonomy including knowledge utilization and evolution. Knowledge utilization delves into the mechanism of memorization, comprehension and application, and creation. Knowledge evolution focuses on the dynamic progression of knowledge within individual and group LLMs. Moreover, we discuss what knowledge LLMs have learned, the reasons for the fragility of parametric knowledge, and the potential dark knowledge (hypothesis) that will be challenging to address. We hope this work can help understand knowledge in LLMs and provide insights for future research.

## 1 Introduction

Knowledge is the cornerstone of intelligence and the continuation of civilization, furnishing us with foundational principles and guidance for navigating complex problems and emerging challenges Davis et al. (1993) ; Choi (2022) . Throughout the extensive history of evolution, we have dedicated our lives to cultivating more advanced intelligence by utilizing acquired knowledge and exploring the frontiers of unknown knowledge McGraw and Harbison-Briggs (1990) ; Han et al. (2021) .

As we know, Large language models (LLMs) are renowned for encapsulating extensive parametric knowledge Roberts et al. (2020) ; Sung et al. (2021) ; Cao et al. (2021a) ; Zhong et al. (2021) ; Kandpal et al. (2023) ; Heinzerling and Inui (2020) ; Petroni et al. (2019) ; Qiao et al. (2023) ; Kritharoula et al. (2023) ; He et al. (2024a) , achieving unprecedented progress in application. However, the knowledge mechanisms in LLMs for learning, storage, utilization, and evolution still remain mysterious Phillips et al. (2021) ; Gould et al. (2023a) ; Wang et al. (2024e) . Extensive works aim to demystify various types of knowledge in LLMs through knowledge neurons Dai et al. (2022) ; Chen et al. (2024a) and circuits Elhage et al. (2021) ; Yao et al. (2024) ; Zou et al. (2024) , yet these efforts, scattered across various tasks, await comprehensive review and analysis.

As shown in Fig 1 , this paper pioneeringly reviews the mechanism across the whole knowledge life cycle. We also propose a novel taxonomy for knowledge mechanisms in LLMs, as illustrated in Fig 2 , which encompasses knowledge utilization at a specific time and knowledge evolution across all periods of LLMs 1 1 1 Knowledge utilization focuses on static knowledge at a specific period, while knowledge evolution explores the long-term dynamic development of knowledge across individual and group LLMs. . Specifically, we introduce preliminaries of this field (§ 2 ) and review the knowledge utilization mechanism from a new perspective (§ 3 ), delve into the fundamental principles for knowledge evolution (§ 4 ). Then, we investigate how to construct more efficient and trustworthy LLMs from the perspective of knowledge mechanism (§ 5 ). Later, We discuss open questions about the knowledge LLMs have and have not acquired (§ 6 ). Finally, we also provide some future directions (§ 7 ) and tools for knowledge mechanism analysis (§ C ). Our contributions are as follows:

• To the best of our knowledge, we are the first to review knowledge mechanisms in LLMs and provide a novel taxonomy across the entire life.

• We propose a new perspective to analyze knowledge utilization mechanisms from three levels: memorization, comprehension and application, and creation .

• We discuss knowledge evolution in individual and group LLMs, and analyze the inherent conflicts and integration in this process.

• We observe that LLMs have learned basic world knowledge. However, the learned knowledge is fragile, leading to challenges such as hallucinations and knowledge conflicts . We speculate that this fragility may be primarily due to improper learning data. Besides, the unlearned dark knowledge will exist long.

##### Comparison with Existing Surveys

Previous interpretability surveys typically aim to investigate various methods for explaining the roles of different components within LLMs from the global and local taxonomy Ferrando et al. (2024) ; Zhao et al. (2024a) ; Luo and Specia (2024) ; Murdoch et al. (2019) ; Rai et al. (2024a) ; Bereska and Gavves (2024) ; Vilas et al. (2024) ; Singh et al. (2024) ; Rai et al. (2024b) . In contrast, this paper focuses on knowledge in LLMs. Hence, our taxonomy, oriented from target knowledge in LLMs, reviews how knowledge is acquired, stored, utilized, and subsequently evolves . Additionally, previous taxonomy mostly explore the explainability during the inference stage (a specific period), while ignoring knowledge acquisition during the pre-training stage and evolution during the post-training stage Räuker et al. (2023) ; Luo et al. (2024b) ; Apidianaki (2023) ; Jiao et al. (2023) ; Räuker et al. (2023) ; Rai et al. (2024c) . Our taxonomy aims to explore the dynamic evolution across all periods from naivety to sophistication in both individual and group LLMs. In contrast to the most similar survey Cao et al. (2024a) that introduces knowledge life cycle, our work focuses on the underlying mechanisms at each stage.

Generally, this paper may help us to explore and manipulate advanced knowledge in LLMs, examine current limitations through the history of knowledge evolution, and inspire more efficient and trustworthy architecture and learning strategy for future models from knowledge mechanism perspective . Note that most hypotheses in this paper are derived from transformer-based LLMs. We also validate the generalizability of these hypotheses across other architectural models and then propose universality intelligence in § B .

## 2 Preliminary

### 2.1 Knowledge Scope

Knowledge is an awareness of facts, a form of familiarity, awareness, understanding, or acquaintance Zagzebski (2017) ; Hyman (1999) ; Mahowald et al. (2023) ; Gray et al. (2024) . It often involves the possession of information learned through experience and can be understood as a cognitive success or an epistemic contact with reality. We denote a diverse array of knowledge as set 𝐊 \mathbf{K} , wherein each element k ∈ 𝐊 k\in\mathbf{K} is a specific piece of knowledge, which can be expressed by various records, e.g., a text record “ The president of the United States in 2024 is Biden ” (denoted as r k r_{k} ).

### 2.2 Definition of Knowledge in LLMs

Given a LLM denoted as ℱ \mathcal{F} , we formulate that ℱ \mathcal{F} master knowledge k k if ℱ \mathcal{F} can correctly answer the corresponding question r k \ t r_{k\backslash t} : t = ℱ ⁡ ( r k \ t ) \displaystyle t=\mathcal{F}\left(r_{k\backslash t}\right) (1) ( t ∈ 𝐓 ) ⇒ ( ℱ ​ masters knowledge ​ k ) , \displaystyle(t\in\mathbf{T})\Rightarrow(\mathcal{F}\ \text{masters knowledge}\ k), t t is the output of a LLM ℱ \mathcal{F} , r k \ t r_{k\backslash t} is a record about knowledge k k that lacks pivot information. Take an example for illustration: r k \ t r_{k\backslash t} is “The president of the United States in 2024 is __” , the pivot information is “Biden”. Note that, r k \ t r_{k\backslash t} can be represented by the above textual statement, captured through a question-answering pair (“ Who is the President of the United States in 2024? ’’, or conveyed by audio, video, image 2 2 2 While audio, video, and image records have been somewhat investigated, they are still relatively unexplored areas and thus are only discussed in § 7.2 and § B.2 . , and other equivalent expressions. The pivot information for r k \ t r_{k\backslash t} can be expressed by various fomarts, which are formulated as 𝐓 = { “Biden” , “Joe Biden” , ⋯ } \mathbf{T}=\left\{\text{``Biden''},\text{``Joe Biden''},\cdots\right\} . If the output t t is an element from the correct answer set 𝐓 \mathbf{T} , we hypothesize that ℱ \mathcal{F} master knowledge k k .

### 2.3 The Architecture of LLMs

An LLM ℱ \mathcal{F} consists of numerous neurons, which work systematically under a specific architecture.

##### Transformer-based architecture.

The prevailing architecture in current LLMs is the Transformer Vaswani et al. (2017) . Specifically, a transformer-based LLM ℱ \mathcal{F} begins with a token embedding, followed by L L layers transformer block, and ends with token unembedding used for predicting answer tokens. Each transformer block layer l l consists of Attention Heads (Attention) and Multilayer Perceptron (MLP): h l + 1 = h l + MLP ​ ( h l + Attention ​ ( h l ) ) , \displaystyle h_{l+1}=h_{l}+\text{MLP}\left(h_{l}+\text{Attention}\left(h_{l}\right)\right), (2) h l h_{l} is the hidden state from l l -th layer.

##### Other architectures.

Other architectures including competitive variants of the transformer, e.g., SSM Gu and Dao (2023) , TTT Sun et al. (2024) and RWKV Peng et al. (2023) , and architectures in computer vision Li et al. (2023c) and multi-modal fields are detailed in § B.1 .

### 2.4 Knowledge Analysis Methods

Knowledge analysis method ℳ \mathcal{M} aims to interpret how LLMs work inside and reveal precise causal connections between specific components and outputs Bereska and Gavves (2024) . Furthermore, if components 𝐂 \mathbf{C} of ℱ \mathcal{F} accurately infer t t through analysis method ℳ \mathcal{M} , it is assumed that the knowledge k k is presented by 𝐂 \mathbf{C} : t = ℳ 𝐂 ⊆ ℱ ​ ( r k \ t , 𝐂 ) , \displaystyle t=\mathcal{M}_{\mathbf{C}\subseteq\mathcal{F}}\left(r_{k\backslash t},\mathbf{C}\right), (3) ( t ∈ 𝐓 ) ⇒ ( 𝐂 ​ represents knowledge ​ k ) , \displaystyle(t\in\mathbf{T})\Rightarrow(\mathbf{C}\ \text{represents knowledge}\ k), The elements in set 𝐂 \mathbf{C} may be individual neurons, MLPs, attention heads, a transformer block layer, or knowledge circuit Yao et al. (2024) . These methods are divided into two categories: observation and intervention Bereska and Gavves (2024) .

##### Observation-based methods.

These methods aim to observe the internal information of ℱ \mathcal{F} , directly projecting the output of component 𝐂 \mathbf{C} into human-understandable forms by E E : t = E 𝐂 ⊆ ℱ ​ ( r k \ t , 𝐂 , ℱ ) , \displaystyle t=E_{\mathbf{C}\subseteq\mathcal{F}}\left(r_{k\backslash t},\mathbf{C},\mathcal{F}\right), (4) E E is a evaluation metric, which can be a probe Räuker et al. (2023) , logit lens nostalgebraist (2020) , or a sparse representation Gao et al. (2024c) . Probe is a meticulously trained classifier, and its classification performance is used to observe the relationship between model’s behavior and the output of 𝐂 \mathbf{C} Belinkov (2022) ; Elazar et al. (2021) ; McGrath et al. (2021) ; Gurnee et al. (2023) . Logit lens usually translate output of 𝐂 \mathbf{C} into vocabulary tokens via token unembedding Geva et al. (2022b) ; Belrose et al. (2023) ; Pal et al. (2023) ; Din et al. (2024) ; Langedijk et al. (2023) . Sparse representation maps the output of 𝐂 \mathbf{C} into a higher-dimensional space with strong sparsity through dictionary learning He et al. (2024b) ; Olshausen and Field (1997) ; Yun et al. (2021) ; Karvonen et al. (2024) , with sparse auto-encoder Sharkey et al. (2022) ; Cunningham et al. (2023) ; Lee et al. (2006) ; Gao et al. (2024a) being a prominent example. The higher-dimensional space represents independent (or monosemantic Bricken et al. (2023) ) and interpretable features more easily Rai et al. (2024c) . The output of 𝐂 \mathbf{C} is the combination Elhage et al. (2022) ; Bricken et al. (2023) of these features.

##### Intervention-based methods.

These methods allow for direct corruptions in LLMs to identify the critical 𝐂 \mathbf{C} via intervention strategies ℐ \mathcal{I} . Note that 𝐂 \mathbf{C} , encompassing various neuron combinations, correlates with specific model behaviors: 𝐂 = ℐ ⁡ ( r k \ t , ℱ ) , \displaystyle\mathbf{C}=\mathcal{I}\left(r_{k\backslash t},\mathcal{F}\right), (5) t = E ⁡ ( r k \ t , 𝐂 , ℱ ) \displaystyle t=E\left(r_{k\backslash t},\mathbf{C},\mathcal{F}\right) ℐ \mathcal{I} is also known as causal mediation analysis Vig et al. (2020) , causal tracing Meng et al. (2022) , interchange interventions Geiger et al. (2022) , activation patching Wang et al. (2023c) ; Zhang and Nanda (2023) , path patching Goldowsky-Dill et al. (2023) , and causal scrubbing techniques LawrenceC et al. (2022) . Specifically, ℐ \mathcal{I} consists of the following three steps. 1) Clean run : ℱ \mathcal{F} generates the correct answer t t based on the input r k \ t r_{k\backslash t} . 2) Corrupted run : corrupt the generation process of ℱ \mathcal{F} in the clean run by introducing noise into the input or neurons Meng et al. (2022) ; Goldowsky-Dill et al. (2023) ; Stolfo et al. (2023) ; Yao et al. (2024) ; Conmy et al. (2023) ; Mossing et al. (2024) ; Lepori et al. (2023) ; Huang et al. (2023a) . 3) Restoration run : recover the correct answer t t by restoring unnoised information from 𝐂 \mathbf{C} Meng et al. (2022) ; Vig et al. (2020) ; Wang et al. (2023c) ; Zhang et al. (2017) ; Nanda (2023) . For intervention-based methods, E E typically refers to the token unembedding used for predicting answer tokens. Under the evaluation metric E E , there exists a causal relationship between 𝐂 \mathbf{C} and specific behavior of LLMs ℱ \mathcal{F} in Eq 5 .

## 3 Knowledge Utilization in LLMs

Inspired by Bloom’s Taxonomy of cognition levels Wilson (2016) ; Bloom et al. (1956) ; Keene et al. (2010) ; Fadul (2009) , we categorize knowledge representation and utilization within LLMs into three levels (as shown in Fig 3 ): memorization, comprehension and application, and creation 3 3 3 Note that we combine analyzing, evaluating, and creating from Bloom’s Taxonomy into one category level (creation) in our taxonomy, as they are difficult to disentangle. Specifically, creation emphasizes the capacity and process of forming novel and valuable things. Analyzing Wilson (2016) , which breaks materials or concepts into parts, is used for creating novel things. Evaluating Wilson (2016) is usually used for assessing the value of new creations. . Note that these mechanistic analyses are implemented via methods in § 2.4 . We further evaluate the applicability, advantages, and limitations of different methods in § 3.4 .

### 3.1 Memorization

Knowledge memorization Schwarzschild et al. (2024) ; Prashanth et al. (2024) aims to remember and recall knowledge in the training corpus, e.g., specific terms (entities), grammar, facts, commonsense, concepts, etc Allen-Zhu and Li (2023a) ; Yu et al. (2023a) ; Mahowald et al. (2023) ; Zhu and Li (2023) ; Allen-Zhu and Li (2023b) ; Allen-Zhu and Li (2024) ; Cao et al. (2024a) . We posit knowledge memorization from Modular Region and Connection Hypothesis by reviewing existing research.