Knowledge Mechanisms in Large Language Models: A Survey and Perspective — part 3/6

Cleaned-text code points [30174, 44842). Bibliography omitted here; source.md includes it.

After pre-training, LLMs are anticipated to refresh their internal knowledge to keep pace with the evolving world during post-training stage . Although LLMs seem to absorb new knowledge through continued learning, follow user instructions via instruct tuning Zhang et al. (2023c) , and align with human values through alignment tuning Ziegler et al. (2019) , Ji et al. (2024) have noted that LLMs intrinsically resist alignment during the post-training phase. In other words, LLMs tend to learn factual knowledge through pre-training, whereas fine-tuning 4 4 4 Fine-tuning includes instruct tuning and alignment tuning Zhao et al. (2023b) . teaches them to utilize it more efficiently Gekhman et al. (2024) ; Zhou et al. (2023a) ; Ovadia et al. (2024) . Ren et al. (2024a) also posit that instruction tuning is a form of self-alignment with existing internal knowledge rather than a process of learning new information. We conjecture that the debate on whether these processes truly introduce new knowledge stems from information conflicts. For example, the conflict between outdated information within LLMs and new external knowledge exacerbates their difficulty in learning new information. To mitigate information conflicts, Ni et al. (2023) propose first forgetting old knowledge then learning new knowledge. Another technique, retrieval-augmented generation (RAG) Huang and Huang (2024) , while avoiding conflicts within internal parameters, still needs to manage conflicts between retrieved external information and LLMs’ internal knowledge Xu et al. (2024b) . RAG also attempt to efficiently and effectively integrate new knowledge across passages or documents using multiple retrieval Yang et al. (2024a) and hippocampal indexing Gutiérrez et al. (2024) . Besides, editing technologies, including knowledge and representation editing, exhibit promising potential for knowledge addition, modification, and erasure. Specifically, knowledge editing Meng et al. (2022) ; Mitchell et al. (2022) ; Cao et al. (2021b) ; Zhang et al. (2024a) ; Wang et al. (2023d) ; Mazzia et al. (2023) aims to selectively modify model parameters responsible for specific knowledge retention, while representation editing Zou et al. (2023) ; Wu et al. (2024) adjusts the model’s conceptualization of knowledge to revise the stored knowledge within LLMs. Note that the other strategy for knowledge editing adds external parameters or memory banks for new knowledge while preserving models’ parameters. We also provide the comparison of the above methods in § A for better understanding.

### 4.2 Group Evolution

Besides individual learning, social interaction plays a pivotal role in the acquisition of new knowledge and is a key driver of human societal development Baucal et al. (2014) ; Levine et al. (1993) . LLMs, also known as agents, collaborate to accomplish complex tasks during group evolution, each bearing unique knowledge that may sometimes contradict each other. Therefore, contrary to individual evolution, group evolution encounters intensified conflicts, such as conflicts in specialized expertise among agents, competing interests, cultural disparities, moral dilemmas, and others . To achieve consensus and resolve conflicts, agents must first clarify their own and others’ goals (beliefs) through internal representations in models Zhu et al. (2024) ; Zou et al. (2023) . Agents then discuss, debate, and reflect on shared knowledge through various communication methods Chan et al. (2024) ; Smit et al. (2024) ; Li et al. (2024e) ; Soltoggio et al. (2024) , e.g., prompt instructions, task and agent descriptions, parameter signals (activation and gradient), and representations of models. However, conformity of agents, which tends to believe the majority’s incorrect answers rather than maintaining their own, hinders conflict resolution during group evolution Zhang et al. (2023a) ; Ma et al. (2024) . Note that the group also struggles with automating moral decision-making when facing moral conflicts. Specifically, agents in the group miss ground truth for moral “correctness” and encounter dilemmas due to changes in moral norms over time Hagendorff and Danks (2023) . Generally, when, what, and how to share knowledge in the communication process to maximize learning efficiency and long-term expectations are still open questions in group evolution.

Through debate and collaboration, groups integrate more knowledge and can surpass the cognition of individual units Liang et al. (2023a) ; Qian et al. (2023) ; Qiao et al. (2024) ; Talebirad and Nadiri (2023) ; Zhang et al. (2023a) . This derives from the assumption that each individual unit can contribute to and benefit from the collective knowledge Soltoggio et al. (2024) ; Xu et al. (2024c) . In addition, “When a measure becomes a target, it ceases to be a good measure” , which implies that optimizing one objective on a single individual will inevitably harm other optimization objectives to some extent. Hence, it is unrealistic for an individual to learn all knowledge compared to group optimization. Interestingly, LLM groups also follow the collaborative scaling law Qian et al. (2024a) , where normalized solution quality follows a logistic growth pattern as scaling agents. Moreover, some works Huh et al. (2024) ; Bereska and Gavves (2024) propose that knowledge tends to converge into the same representation spaces among the whole artificial neural models group with different data, modalities, and objectives.

### 4.3 Comparison of Different Evolution Strategies

Individuals and groups achieve dynamic intelligence primarily through two strategies: updating internal parametric knowledge Zhou et al. (2023a) ; Qiao et al. (2024) and leveraging external knowledge 5 5 5 Leveraging external knowledge includes using prompts Xie et al. (2024) , ICL, and RAG. Huang and Huang (2024) ; Xie et al. (2024) . These two strategies are usually used together in applications Yang et al. (2024b) .

Updating internal parametric knowledge necessitates high-quality data for parameter adjustments Vashishtha et al. (2024) ; Cao et al. (2024a) . Data proves pivotal when fine-tuning models to acquire new knowledge. Ovadia et al. (2024) also posit that the continued training of LLMs via unsupervised tuning generally exhibits suboptimal performance when it comes to acquiring new knowledge. Note that updating internal parametric knowledge requires resolving conflicts among internal parameters. The crux of effective internal knowledge updating lies in preserving the consistency of the model’s parameter knowledge before and after tuning. In contrast, leveraging external knowledge requires managing conflicts within the external knowledge itself 6 6 6 Inconsistencies in external information are common, as external documents often contain conflicting data, particularly in contexts for RAG. as well as conflicts between external and internal knowledge Xu et al. (2024b) ; Liu et al. (2024a) . Besides, parametric knowledge compresses extensive information, promoting grokking and enhancing generalization Wang et al. (2024a) . In contrast, leveraging external knowledge avoids high training costs but necessitates substantial maintenance and retrieval costs for every user query. Therefore, the combination of these two strategies is promising. An attempt for combination Yang et al. (2024b) suggests employing RAG for low-frequency knowledge and parametric strategy for high-frequency knowledge.

## 5 Application of Knowledge Mechanism

The mechanism analysis of knowledge utilization and evolution may provide an avenue to construct more efficient and trustworthy models in practice.

### 5.1 Efficient LLMs

Researchers have been working to reduce the cost of training and inference for LLMs through various optimization strategies, including architecture Ainslie et al. (2023) ; Fedus et al. (2022) , data quality Kaddour (2023) , parallelization Qi et al. (2024) , generalization theory Zhang et al. (2024d) , hardware Dey et al. (2023) , scaling laws Hoffmann et al. (2022) , optimizer Liu et al. (2023a) , etc. The underlying knowledge mechanisms offer LLMs new potential for efficiently storing, utilizing, and evolving knowledge.

For knowledge storage and utilization in LLMs, knowledge (memory) circuit provides the theory to decompose the knowledge computations of an LLM into smaller, recurring parts Yang et al. (2024b) . These smaller parts guide the determination of which types of knowledge should be encoded into parameters. Therefore, Memory 3 \text{Memory}^{3} Yang et al. (2024b) designs an explicit memory mechanism for Transformer-based LLMs, alleviating the burden of parameter size. Specifically, Memory 3 \text{Memory}^{3} designs external information, explicit memory, and implicit memory for different usage frequencies, reducing writing and reading costs. For knowledge evolution , the knowledge mechanism analysis inspires editing and model merging . The details of editing technologies can be found in § 4.2 . Model merging technologies 7 7 7 Model merging also includes methods that directly interpolation Goddard et al. (2024) or randomly fusing Yu et al. (2023b) , ignoring parameter directions. These methods are naive and are not the focus of our discussion here. leverage parameter directions to combine multiple task-specific models into a single multitask model without performing additional training rather than training from scratch. For instance, Task Arithmetic Ilharco et al. (2023) identifies the weight directions of task capabilities in different models, and then integrates a more powerful model by arithmetic operations on weight directions. TIES Yadav et al. (2023) resolves parameters directions conflicts, and merges only the parameters that are in alignment with the final agreed-upon sign. Akiba et al. (2024) further propose evolutionary optimization of model merging, which automatically discovers effective combinations of open-source models, harnessing their group intelligence without requiring extensive training data or computational resources. Besides, the Lottery Ticket Hypothesis Frankle and Carbin (2019) provides a cornerstone for model compression , generalizing across various datasets, optimizers, and model architectures Morcos et al. (2019) ; Chen et al. (2021) . However, model compression often limits the success of editing and model merging Kolbeinsson et al. (2024) . This phenomenon poses challenges for practical implementations, highlighting the need for more effective strategies.

### 5.2 Trustworthy LLMs

Numerous studies investigate the underlying causes of security risks Reuel et al. (2024) ; Ren et al. (2024b) ; Li et al. (2024a) ; Bengio (2024) ; Bengio et al. (2024) ; Dalrymple et al. (2024) . In particular, Wei et al. (2023) delve into the safety of LLM and reveal that the success of jailbreak is mainly due to the distribution discrepancies between malicious attacks and training data. Geva et al. (2022b) and Wang et al. (2024b) further discover that some parameters within LLMs, called toxic regions, are intrinsically tied to the generation of toxic content. Ji et al. (2024) even conjecture that LLMs resist alignment. Therefore, traditional aligned methods, DPO Rafailov et al. (2023) and SFT, seem to merely bypass toxic regions Lee et al. (2024) ; Wang et al. (2024b) , making them susceptible to other jailbreak attacks Zhang et al. (2023d) .

Inspired by the knowledge mechanism analysis in LLMs, a promising trustworthy strategy may be designing architecture and training process during the pre-training phase to encouraging modularity Liu et al. (2024c) ; Liu et al. (2023b) , sparsity Chughtai et al. (2023) , and monosemanticity Bricken et al. (2023) ; Jermyn et al. (2022) , which make the reverse engineering process more tractable Jermyn et al. (2022) ; Bricken et al. (2023) ; Liu et al. (2024c) ; Tamkin et al. (2023) . Yet, maintaining sparsity for a vast amount of world knowledge requires substantial resources, and whether monosemantic architecture can support advanced intelligence remains elusive. Besides, machine unlearning Nguyen et al. (2022) ; Tian et al. (2024) ; Yao et al. (2023a) aims to forget privacy or toxic information learned by LLMs. However, these unlearning methods suffer overfitting, forgetting something valuable due to the difficulty of disentangling verbatim memorization and general capabilities Huang et al. (2024c) ; Blanco-Justicia et al. (2024) . Another alternative technique is knowledge editing , precisely modifying LLMs using few instances during the post-training stage Mazzia et al. (2023) ; Yao et al. (2023b) ; Wang et al. (2023d) ; Hase et al. (2024) ; Qian et al. (2024b) . Extensive experiments demonstrate that knowledge editing has the potential to detoxify LLMs Yan et al. (2024) . Specifically, Wu et al. (2023a) and Geva et al. (2022b) deactivate the neurons related to privacy information and toxic tokens, respectively. Wang et al. (2024b) identify and then erases toxic regions in LLMs. However, knowledge editing also introduces side effects, such as the inability of the modified knowledge to generalize to multi-hop tasks Zhong et al. (2023) ; Li et al. (2023d) ; Cohen et al. (2023) ; Kong et al. (2024) and the potential to impair the model’s general capabilities Gu et al. (2024) ; Qin et al. (2024) . Therefore, recent efforts focus on representation editing instead of editing parameters in knowledge editing Zou et al. (2023) ; Turner et al. (2023) ; Zhou et al. (2023b) ; Zhu et al. (2024) . These representations (hidden states) within LLMs can trace and address a wide range of safety-relevant problems, including honesty, harmlessness, and power seeking. Later, Wu et al. (2024) develop a family of representation finetuning methods to update new knowledge. Zou et al. (2024) propose circuit-breaking Li et al. (2023b) , directly controlling the representations that are responsible for harmful outputs. However, these representation editing strategies require meticulous hyperparameter tuning for each task. More efficient optimization methods are needed to align with computational or temporal constraints.

## 6 Discussion

In this section, we discuss some open questions and seek to explore their essence and underlying principles. Specifically, we discuss what knowledge LLMs have learned in § 6.1 , examine the fragility of the learned knowledge in application in § 6.2 , analyze the dark knowledge not yet learned by machines or humans in § 6.3 , and explore how LLMs can expand the boundaries of unknown knowledge from interdisciplinary perspectives § 6.4 .

### 6.1 What Knowledge Have LLMs Learned?