Knowledge Mechanisms in Large Language Models: A Survey and Perspective — part 2/6

Cleaned-text code points [14825, 30174). Bibliography omitted here; source.md includes it.

This modular region hypothesis simplifies knowledge representation in transformer-based models into isolated modular region, e.g., MLPs or attention heads. Knowledge is encoded via MLPs. Geva et al. (2021) posit that MLPs operate as key-value memories and each individual key vector corresponds to a specific semantic pattern or grammar . Based on the above finding, Geva et al. (2022b) ; Geva et al. (2022a) reverse engineer the operation of the MLPs layers and find that MLPs can promote both semantic (e.g., measurement semantic including kg, percent, spread, total, yards, pounds, and hours) and syntactic (e.g., adverbs syntactic including largely, rapidly, effectively, previously, and normally) concepts in the vocabulary space. Miller and Neo (2024) find a single MLP neuron (in GPT-2 Large) capable of generating “an” or “a”. Subsequently, fact Dai et al. (2022) ; Meng et al. (2022) and commonsense knowledge Gupta et al. (2023) are found. Advanced language-specific neurons Tang et al. (2024) , linguistic regions Zhao et al. (2023a) , entropy neurons Stolfo et al. (2024) , query-relevant neurons Chen et al. (2024b) , abstract conceptual Wang et al. (2024f) and unsafe Wang et al. (2024b) ; Wu et al. (2023a) knowledge, are also observed in MLPs. In addition to MLP, knowledge is also conveyed by attention heads Geva et al. (2023) ; Gould et al. (2023b) . Hoover et al. (2020) explain the knowledge each attention head has learned. Specifically, attention heads store evident linguistic features, positional information , and so on. Besides, fact knowledge Yu et al. (2023c) ; Li et al. (2023a) and bias Hoover et al. (2020) are mainly convey by attention heads. Jiang et al. (2024b) further observe that LLMs leverage self-attention to gather information through certain tokens in the contexts, which serve as clues, and use the value matrix for associative memory. Later, Zhu et al. (2024) also find that attention heads can simulate mental stat e and activate “Theory of Mind” (ToM) capability.

However, Hypothesis 1 ignores the connections between different regions. Inspired by advancements in neuroscience de Schotten et al. (2022) , Hypothesis 2 asserts that the connection of different components integrates knowledge, rather than the isolated regions in Hypothesis 1.

Geva et al. (2023) outline the encoding of factual knowledge (e.g., “The capital of Ireland is Dublin”) through the following three steps: (1) subject (Ireland) information enrichment in MLPs, (2) the relation (capital of) propagates to the last token, (3) object (Dublin) is extracted by attention heads in later layers. This claim is supported by Li et al. (2024d) . Similarly, Lv et al. (2024) conclude that task-specific attention head may move the topic entity to the final position of the residual stream, while MLPs conduct relation function. Moreover, the recent prominent knowledge circuit framework Nainani (2024) ; Yao et al. (2024) ; He et al. (2024b) ; Elhage et al. (2021) ; Marks et al. (2024) advocates leveraging a critical computational subgraph among all components to explore internal knowledge within LLM parameters. The competencies for indirect object identification and color object tasks are discovered to be embedded in specialized knowledge circuits Conmy et al. (2023) ; Wang et al. (2023c) ; Merullo et al. (2023a) ; Yu et al. (2024c) . Lan et al. (2024) also identify number-related circuits that encode the predictive ability of Arabic numerals, number words, and months. More importantly, experimental evidence demonstrates that various types of knowledge, including linguistic, commonsense, factual, and biased information, are encapsulated in specific knowledge circuits Yao et al. (2024) . Interestingly, knowledge encoded by specific circuits can rival or even surpass that of the entire LLM. This may be because knowledge circuits memorized the relevant knowledge, while noise from other components might impede the model’s performance on these tasks.

### 3.2 Comprehension and Application

Knowledge comprehension and application focus on demonstrating the understanding of memorized knowledge and then solving problems in new situations, e.g., generalization on out-of-domain tasks Wang et al. (2024a) , reasoning Hou et al. (2023) and planning McGrath et al. (2021) . Merrill et al. (2023) denote the transition from memorization to comprehension and application as grokking, and suggest that the grokking derives from two largely distinct subnetworks competition. Intuitively, only knowledge that is correctly memorized Prashanth et al. (2024) in § 3.1 can be further applied to solving complex tasks. Therefore, we posit the following Reuse Hypothesis from two knowledge memorization perspectives.

From the Modular Region Perspective, knowledge utilization reuses some regions. These regions might include a few neurons, attention heads, MLPs, a transformer layer, or partial knowledge circuits. Generally, basic knowledge (position information, n-gram pattern, syntactic features) tends to be stored at earlier layers, while sophisticated knowledge (mental state, emotion, and abstract concept, e.g., prime number, Camelidae, and safety) is located at later layers Zhu et al. (2024) ; Jin et al. (2024a) ; Wang et al. (2024b) ; Wang et al. (2024f) ; Men et al. (2024) ; Kobayashi et al. (2023) . Therefore, neurons of earlier layers related to basic knowledge tend to be reused Kang and Choi (2023) ; Zhao et al. (2024a) ; Kandpal et al. (2023) . Various math reasoning tasks also utilize the attention mechanism in initial layers to map input information to the final token positions, subsequently generating answers using a set of MLPs in later layers Stolfo et al. (2023) ; Hanna et al. (2023) ; Langedijk et al. (2023) . Besides, some specific function regions are also reused . Specifically, retrieval heads Li et al. (2023a) are reused for Chain-of-Thought (CoT) reasoning and long-context tasks. These retrieval heads are found in 4 model families, 6 model scales, and 3 types of fine-tuning. Subsequently, induction heads, identified in Llama and GPT, are claimed to be reused for in-context learning (ICL) tasks Olsson et al. (2022) ; Crosbie and Shutova (2024) . Attention heads can map country names to their capitals in capital city-related tasks Lv et al. (2024) . Language-specific neurons (in Llama and BLOOM) are responsible for multiple language related tasks, such as English, French, Mandarin, and others Tang et al. (2024) . Zhao et al. (2023a) further reveal linguistic regions (in Llama) correspond to linguistic competence, which is the cornerstone for performing various tasks. Later, function regions related to the process of math reasoning are also discovered in LLMs. For instance, the last layer of GPT-2 (trained from scratch) has been observed to exhibit mathematical reasoning abilities across various math questions Ye et al. (2024) . From the Connection Perspective, knowledge utilization shares partial knowledge circuits. For instance, similar tasks share subgraphs (computational circuits) with analogous roles Lan et al. (2024) . Besides, knowledge circuits (in GPT2) are reused to solve a seemingly different task, e.g., indirect object identification and colored objects tasks Merullo et al. (2023a) . Wang et al. (2024a) further observe that two-hop composition reasoning tasks reuse the knowledge circuits from the first hop. Yao et al. (2024) also believe that this reuse phenomenon exists in factual recall and multi-hop reasoning. Specifically, sub-circuits are reused in similar factual knowledge, such as tasks related to “city_in_country”, “name_birth_place”, and “country_language”. Besides, Dutta et al. (2024) demystify LLMs how to perform CoT reasoning, i.e., Llama facilitates CoT tasks via multiple parallel circuits enjoying significant intersection.

### 3.3 Creation

Knowledge creation Runco and Jaeger (2012) ; Sternberg (2006) emphasizes the capacity and process of forming novel and valuable things, rather than the existing ones (i.e., LLMs have seen) discussed in § 3.1 and § 3.2 . The creations encompass two levels: 1) LLMs create new terms following the current world’s principles comprehended by LLMs, such as new proteins Shin et al. (2021) , molecules Bagal et al. (2022) ; Fang et al. (2023) ; Edwards et al. (2022) , code DeLorenzo et al. (2024) , video Kondratyuk et al. (2023) , models Zheng et al. (2024) , names for people and companies, written stories Pépin et al. (2024) ; Gómez-Rodríguez and Williams (2023) ; Buz et al. (2024) , synthetic data Stenger et al. (2024) ; Mumuni et al. (2024) ; Abufadda and Mansour (2021) , etc. These novel items operate according to the existing rules, e.g., law of conservation of energy, reasoning logic Wang et al. (2024a) , or principles of probability theory. 2) LLMs may generate new rules, such as mathematical theorems, and the resulting terms will operate according to the new rules. We posit that the knowledge creation of LLMs may derive from the Extrapolation Hypothesis.

The expression of knowledge is diverse; some knowledge is inherently continuous. Therefore, it is difficult, if not impossible, to represent certain knowledge using discrete data points Spivey and Michael (2007) ; Penrose () ; Markman (2013) . LLMs utilize insights into the operational principles of the world to extrapolate additional knowledge from known discrete points, bridging gaps in knowledge and expanding our understanding of the world Heilman et al. (2003) ; Douglas et al. (2024) ; Park et al. (2023b) ; Kondratyuk et al. (2023) . Drawing inspiration from research on human creativity Haase and Hanel (2023) , the physical implementation of knowledge extrapolation relies on the plasticity of neurons Mukherjee and Chang (2024) . Specifically, plasticity refers to LLMs changing activations and connectivity between neurons according to the input Coronel-Oliveros et al. (2024) .

However, from statistical perspective, the intricate connections and activations between neurons, though not infinite, resist exhaustive enumeration. In terms of value, not all creations are valuable. Obtaining something valuable with an exceedingly low probability is impractical, as even a monkey could theoretically print Shakespeare’s works. How do LLMs ensure the probability of generating valuable creations? What are the mechanisms underlying the novelty and value of creation? A prevalent conjecture posits that novelty is generated through the random walk Sæbø and Brovold (2024) . However, intuitively, current LLMs themselves seem unable to evaluate the value of creations due to architectural limitations Chakrabarty et al. (2024) . Because, once the next token is generated, there is no intrinsic mechanism for accepting or rejecting the creations. This hinders the evaluation of the usefulness and value of proposed novelties, as humans do, by bending, blending, or breaking biases Sæbø and Brovold (2024) . Some works assume that each token is indeed valuable and meets long-term expectations. However, the well-known hallucination problem Xu et al. (2024d) of LLMs refutes this assumption. Besides, the transformer architecture struggles with long context Li et al. (2024b) , despite the existence of many variants for addressing this issue Huang et al. (2023c) ; Liu et al. (2024b) . More importantly, MLPs of Transformer may also work contrary to creativity, i.e., the increased attentions narrow the conditional distribution for token prediction Sæbø and Brovold (2024) .

### 3.4 Comparison of Different Mechanism Analysis Methods

The above four Hypotheses are achieved by Observation-based and Intervention-based methods. These two methods are typically combined to trace knowledge in LLMs Mossing et al. (2024) ; Ghandeharioun et al. (2024) . Most knowledge analysis methods are architecture-agnostic and can be adapted to various models.

Each method is suitable for different scenarios. Specifically, the Modular Region Hypothesis can be analyzed using either Observation-based or Intervention-based methods. In contrast, the Connection Hypothesis, which examines inter-regional connectivity, generally necessitates Intervention-based methods. However, the results of knowledge mechanism analysis depend heavily on different methods and are sensitive to evaluation metrics and implementation details Schwettmann et al. (2023b) . Hence, Huang et al. (2024b) propose a dataset, RAVEL, to quantify the comparisons between a variety of existing interpretability methods. They suggest that methods with supervision are better than methods with unsupervised featurizers. Later, Zhang and Nanda (2023) further systematically examine the impact of methodological details in intervention-based methods. For corrupted run, they recommend Symmetric Token Replacement (e.g., “The Eiffel Tower” → \rightarrow “The Colosseum”) Sharma et al. (2024) ; Vig et al. (2020) instead of Gaussian Noising Meng et al. (2022) , which disrupts the model’s internal mechanisms. For metric E E , both logit lens and probe can be employed to trace factual knowledge Meng et al. (2022) , where the target output is typically few tokens. In this scenario, Zhang and Nanda (2023) advocate using the logit lens over probes for evaluation metric E E due to its fine-grained control over localization outcomes. Moreover, probe is capable of exploring abstract knowledge and abilities, such as theory of mind or mental states Zhu et al. (2024) ; Ye et al. (2024) ; Jin (2024) , where the target output requires multiple tokens to express. Jin (2024) suggest that deeper probes are more (generally) more accurate.

## 4 Knowledge Evolution in LLMs

Knowledge in LLMs should evolve with changes in the external environment. We introduce the Dynamic Intelligence Hypothesis for knowledge evolution in individuals and groups.

### 4.1 Individual Evolution

Immersed in a dynamic world, individuals mature through an iterative process of memorization, forgetting, error correction, and deepening understanding of the world around them. Similarly, LLMs dynamically encapsulate knowledge into parameters through the process of conflict and integration.

In the pre-training phase , LLMs start as blank slates, facilitating easier acquisition for new knowledge Allen-Zhu and Li (2024) . Consequently, numerous experiments demonstrate that LLMs accumulate vast amounts of knowledge during this stage Cao et al. (2024b) ; Zhou et al. (2023a) ; Kaddour et al. (2023) ; Naveed et al. (2023) ; Singhal et al. (2022) . Akyürek et al. (2022) delve further into identifying which training examples are instrumental in endowing LLMs with specific knowledge. However, contradictions during the pre-training stage may induce conflicts among internal parametric knowledge. On the one hand, the false and contradictory information in training corpus propagate and contaminate related memories in LLMs via semantic diffusion, introducing broader detrimental effects beyond direct impacts Bian et al. (2023) . On the other hand, LLMs tend to prioritize memorizing more frequent and challenging facts, which can result in subsequent facts overwriting prior memorization, significantly hindering the memorization of low-frequency facts Lu et al. (2024) . In other words, LLMs struggle with balancing and integrating both low and high-frequency knowledge.