Knowledge Mechanisms in Large Language Models: A Survey and Perspective — part 4/6

Cleaned-text code points [44842, 60498). Bibliography omitted here; source.md includes it.

Critics question whether LLMs truly have knowledge or if they are merely mimicking Schwarzschild et al. (2024) , akin to the “Stochastic Parro” Bender et al. (2021) and “Clever Hans” Shapira et al. (2024) . We first review the doubts from the following three levels through observation phenomena : 1) Memorization: LLMs primarily rely on positional information over semantic understanding Li et al. (2022) to predict answers. Additionally, LLMs may generate different answers for the same question due to different expressions. 2) Comprehension and application: Allen-Zhu and Li (2023b) argue that LLMs hardly efficiently apply knowledge from pre-training data, even when such knowledge is perfectly stored and fully extracted from LLMs. Therefore, LLMs struggle with various reasoning tasks Wu et al. (2023b) ; Nezhurina et al. (2024) ; Gutiérrez et al. (2024) as well as the reversal curse Berglund et al. (2023) . Besides, LLMs are not yet able to reliably act as text world simulators and encounter difficulties with planning Wang et al. (2024d) . 3) Creation: Although LLMs are capable of generating new terms, their quality often falls below that created by humans Raiola (2023) . Even though LLMs possess knowledge, some critics argue that current analysis methods may only explain low-level co-occurrence patterns, not internal mechanisms. The primary criticism asserts that the components responsible for certain types of knowledge in LLM fail to perform effectively in practical applications Hase et al. (2023) . In addition, the components responsible for specific knowledge within LLMs vary under different methods. For these criticisms, Chen et al. (2024g) ; Chen et al. (2024e) propose degenerate neurons and posit that different degenerate components indeed independently express a fact. Chen et al. (2024f) delineate the differences in the mechanisms of knowledge storage and representation, proposing the Query Localization Assumption to response these controversies. Zhu and Li (2023) further observe that knowledge may be memorized but not extracted due to the knowledge not being sufficiently augmented (e.g., through paraphrasing, sentence shuffling) during pretraining. Hence, rewriting the training data to provide knowledge augmentation and incorporating more instruction fine-tuning data in the pre-training stage can effectively alleviate the above challenges and criticisms.

Despite considerable criticism, the mainstream view Didolkar et al. (2024) ; Jin and Rinard () ; Jin (2024) is that current LLMs may possess basic world knowledge via memorization but hardly master underlying principles for reasoning and creativity . In other words, LLMs master basic knowledge via memorization (discussed in § 3.1 ). Although LLMs possess the foundational ability to comprehend and apply knowledge (discussed in § 3.2 ), exhibiting plausible and impressive reasoning capabilities. Current LLMs still struggle with reasoning and planning in complex tasks due to the fragility of knowledge in LLMs (elaborated in § 6.2 ). These reasoning and planning abilities usually require to be induced through techniques such as ICL and CoT. Unfortunately, current LLMs are nearly incapable of creation due to the architectural limitations (discussed in § 3.3 ). Therefore, some scholars explore various architectural choices (e.g., Mamba Gu and Dao (2023) ) and training procedures. Besides, recent research attempts to manipulate neurons, knowledge circuits, or representations Allen-Zhu and Li (2023b) ; Zou et al. (2023) ; Wu et al. (2024) ; Li et al. (2023a) to explore more knowledge and awaken the reasoning and planning capabilities of LLMs.

Remarks : LLMs have learned basic knowledge of the world by momorization . However, the learned knowledge is fragile, leading to challenges in knowledge comprehension and application . Unfortunately, due to architectural limitations, current LLMs struggle with creation .

### 6.2 Why Is Learned Knowledge Fragile?

The knowledge learned by LLMs is fragile , leading to challenges in application including hallucination, knowledge conflicts, failed reasoning, and safety risk 8 8 8 The secure risk is elaborated in § 5.2 . . Hallucination denotes content generated by LLMs that diverges from real-world facts or inputs Huang et al. (2023b) ; Xu et al. (2024d) ; Farquhar et al. (2024) ; Chen et al. (2024d) . On the one hand, factuality hallucination underscores the disparity between generated content and real-world knowledge. On the other hand, faithfulness hallucination describes the departure of generated content from user instructions or input context, as well as the coherence maintained within the generated content. Knowledge Conflict inherently denotes inconsistencies in knowledge Xu et al. (2024b) ; Kortukov et al. (2024) . On the one hand, internal memory conflicts within the model cause LLMs to exhibit unpredictable behaviors and generate differing results to inputs which are semantically equivalent but syntactically distinct Xu et al. (2024b) ; Wang et al. (2023a) ; Feng et al. (2023b) ; Raj et al. (2022) . On the other hand, context-memory conflict emerges when external context knowledge contradicts internal parametric knowledge Xu et al. (2024b) ; Mallen et al. (2023) .

We posit that these challenges mainly derive from improper learning data. Specifically, hallucination is introduced by data Kang and Choi (2023) ; Weng (2024) ; Zhang et al. (2024c) , heightened during the pre-training Brown et al. (2020) ; Chiang and Cholak (2022) , alignment Azaria and Mitchell (2023) ; Ouyang et al. (2022) , and deficiencies in decoding strategies Fan et al. (2018) ; Chuang et al. (2023) ; Shi et al. (2023) . Internal memory conflict can be attributed to training corpus bias Wang et al. (2023b) , and exacerbated by decoding strategies Lee et al. (2022b) and knowledge editing. Context-memory conflict arises mainly from the absence of accurate knowledge during training, necessitating retrieval from databases and the Web. Failed reasoning usually arises from improper data distribution . Specifically, knowledge may be memorized but not extractable or applicable without sufficient augmentation (e.g., through paraphrasing, sentence shuffling) during pre-training Zhu and Li (2023) . Antoniades et al. (2024) also delve into the mechanism between parametric knowledge and learning data, demonstrate that training data distribution qualitatively influences generalization behavior Jiang et al. (2024a) . Wang et al. (2024a) further suggest that improper data distribution in the corpus causes LLMs to lack essential reasoning components, such as the bridge layer for two-hop reasoning. Similar mechanism analysis also supports the above conclusion, indicating that hallucinations arise from a lack of mover heads Yao et al. (2024) ; Yu et al. (2024b) , while knowledge conflicts stem from circuit competition failure in the last few layers Lv et al. (2024) ; Merullo et al. (2023b) ; Hase et al. (2023) ; Ju et al. (2024) ; Jin et al. (2024b) . Additionally, data quantity is crucial for knowledge robustness. Specifically, LLMs can systematically learn comprehensive understandings of the world from extensive datasets, while little data during post-training stage may compromise the robustness of knowledge representation. This assumption is confirmed by numerous failures of post-training. For example, SFT exacerbates hallucinations Gekhman et al. (2024) ; Kang et al. (2024) , and knowledge editing amplifies knowledge conflicts Li et al. (2023d) ; Yang et al. (2024c) . Note that safety issues usually caused by the distribution of unseen data (adversarial input) Wei et al. (2023) ; Li et al. (2024c) , which is elaborated in § 5.2 .

Remarks : Improper learning caused by data distribution and quantity might be the fundamental and primary cause.

### 6.3 Does Difficult-to-Learn “Dark Knowledge” Exist?

The distribution and quality of data are vital for knowledge acquisition and robust operation within the model (machine). Imagine an ideal scenario where we have access to all kinds of data to train the machine. The data includes all possible modalities, such as text, image, audio, video, etc. Models can also interact with each other and the external environment. In this long-term development, will there still be unknown dark knowledge for intelligence to human or model (machine)?

We hypothesize that there will still exist dark knowledge for intelligence in the future. As shown in Fig 4 , dark knowledge describes knowledge unknown to human or machine from the following three situations: 1) knowledge unknown to human & known to machine (UH, KM). Machines leverage vast amounts of data to explore internal patterns, whereas humans struggle with processing such data due to physiological limitations on data processing capacity and computational limits Burns et al. (2023) ; McAleese et al. (2024) . (UH, KM) includes gene prediction, intelligent transportation systems, and more. Specifically, the structural elucidation of proteins remains mysterious to humans for a long time. Cryo-electron microscopy, through capturing millions of images, first reveals the three-dimensional structures of proteins. Now, neural models can directly predict protein properties with high efficiency and accuracy Pak et al. (2023) . 2) knowledge known to human & unknown to machine (UH, KM). On the one hand, some scholars claim that machine can possess a “Theory of Mind” capability Zhu et al. (2024) and emotions Normoyle et al. (2024) . On the other hand, critics contend that machine lacks sentience Alvero and Peña (2023) and merely probabilistically generates tokens. The causes, extent, and dynamics of these emotions and sentience (like hunger, happiness, and loneliness) are subtle and intricate, making precise mathematical modeling by the machine exceptionally challenging. Specifically, different factors are tightly coupled, making it nearly impossible to disentangle clear input-output relationships as with well-defined factual knowledge. The sentient knowledge also exhibits chaotic behavior Li et al. (2020) ; Debbouche et al. (2021) , being highly sensitive to initial conditions, where small changes can lead to vastly different outcomes Segretain et al. (2020) . Therefore, opponents argue that no matter how many parameters machine possesses, it cannot learn all the knowledge that human has mastered. 3) knowledge unknown to human & unknown to machine (UH, UM) is beyond our cognition, e.g., the uncertainty in quantum mechanics and the origin of the universe. Generally, Dark knowledge extends beyond current data and model architectures Tseng et al. (2024) . (UH, UM) necessitates human-machine collaboration. Yet, there is no definitive conclusion on whether (UH, KM) and (KH, UM) will be solved by model architecture, training data, and computational resources. Note that plain knowledge known to human and machine in Fig 4 encompasses well-defined historical events, mathematical theorems, physical laws, etc.

Remarks : Dark knowledge may persist for a long time and requires human-machine collaboration to explore.

### 6.4 How to Explore More Knowledge from Interdisciplinary Inspiration?

How can LLMs continuously narrow the boundaries of dark knowledge and achieve higher level intelligence by leveraging the human experience of perpetual knowledge exploration throughout history? We may draw inspirations from the following interdisciplinary studies.

Neuroscience studies the structure and function of the brain at molecular, cellular, neural circuit, and neural network levels Squire et al. (2012) . Generally, both mechanism analysis in LLMs and neuroscience utilize observation and intervention methods to investigate the basic principles of knowledge learning and memory, decision-making, language, perception, and consciousness. The biological signals of the human brain and the internal activation signals in LLMs are capable of reciprocal transformation Caucheteux et al. (2023) ; Feng et al. (2023a) ; Mossing et al. (2024) ; Flesher et al. (2021) . Benefiting from advancements in neuroscience Jamali et al. (2024) ; de Schotten et al. (2022) ; Lee et al. (2022a) , mechanism analysis in LLMs has identified analogous function neurons and regions Zhao et al. (2023a) , and knowledge circuits Yao et al. (2024) . Besides, leveraging plasticity theory in neuroscience, LLMs explain the underlying technical support for intelligence Sæbø and Brovold (2024) . In the future, mechanism analysis of LLMs may draw inspirations from neuroscience, guiding the next generation of artificial intelligence in organizing neural frameworks and in the storage and utilization of knowledge Ren and Xia (2024) ; Momeni et al. (2024) ; Yang et al. (2024b) .

Cognitive Science focuses on the mind and its processes Kolak et al. (2006) ; Baronchelli et al. (2013) , which include language, perception, memory, attention, reasoning, emotion and mental state. Although cognitive science and neuroscience overlap in their research content, cognitive science focuses more on abstract knowledge such as mental states and emotions rather than specific knowledge. Therefore, Zhu et al. (2024) track beliefs of self and others (formulated as “Theory of Mind”) in LLMs from the psychological perspective within cognitive science. Wang et al. (2022) further observe social-cognitive skill in multi-agent communication and cooperation. Generally, there is potential to explore advanced cognitive capabilities in LLMs from the perspective of cognitive science Vilas et al. (2024) .

Psychology is the scientific study of mind and behavior, which include both conscious and unconscious phenomena, and mental processes such as thoughts, feelings, and motives. Benefiting from decades of research in human psychology, machine psychology aims to uncover mechanisms of decision-making and reasoning in LLMs by treating them as participants in psychological experiments Hagendorff (2023) . Machine psychology may delve into mysteries of social situations and interactions shaping machine behavior, attitudes, and beliefs Park et al. (2023a) . Besides, group psychology paves an auspicious path for exploring dynamics such as debates and collaboration among LLMs (agents). For instance, Dunning–Kruger effect Mahmoodi et al. (2013) ; Brown and Esterle (2020) in cognitive psychology filed describes that individuals with limited competence in a particular domain overestimate their abilities, and vice versa. This phenomenon may guide the final vote in group debates and discussions. Promisingly, psychology of learning can be applied to study prompt designs, boost learning efficiency, improve communication strategies, and develop feedback mechanisms for LLMs Leon (2024) .

Education is the transmission of knowledge, skills, and character traits and manifests in various forms. Inspired by education in humans, Zhang et al. (2024a) categorize knowledge acquisition in LLMs into three distinct phases: recognition, association, and mastery. Besides, education instructs humans managing various types of conflicts: identifying inconsistencies in external information (inter-context conflict), deciding between external sources and internal memory (context-memory conflict), resolving memory confusion (internal memory conflict), and addressing cultural conflicts. The above knowledge conflicts and integration also exist in knowledge evolution of LLMs across individuals and groups Dan et al. (2023) . Fortunately, education facilitates humans in learning to learn. Can LLMs similarly self-evolve to continuously adapt to societal changes and requirements?