Knowledge Mechanisms in Large Language Models: A Survey and Perspective — part 6/6

Cleaned-text code points [75725, 86737). Bibliography omitted here; source.md includes it.

is a mechanism in neural networks, especially in models like Transformers, that captures dependencies between different positions within a sequence. It works by transforming each input element into Query ( Q Q ), Key ( K K ), and Value ( V V ) vectors, computing attention scores between elements, and then calculating a weighted sum of values based on these scores. Specifically, for an input sequence represented as matrix X X , the transformations are as follows: Q \displaystyle Q = X ​ W Q , \displaystyle=XW^{Q}, (7) K \displaystyle K = X ​ W K , \displaystyle=XW^{K}, V \displaystyle V = X ​ W V , \displaystyle=XW^{V}, where W Q W^{Q} , W K W^{K} , and W V W^{V} are learned projection matrices. The attention scores are computed using the scaled dot-product attention mechanism: H = Attention ​ ( Q , K , V ) = Softmax ​ ( Q ​ K T d k ) ​ V , \displaystyle H=\text{Attention}(Q,K,V)=\text{Softmax}\left(\frac{QK^{T}}{\sqrt{d_{k}}}\right)V, (8) where d k d_{k} is the dimensionality of the Key vectors. This allows the model to focus on different parts of the sequence adaptively, making it effective for tasks like natural language processing where understanding long-range dependencies is important.

##### Variants of Transformer

Variants of the Transformer also achieve success. For instance, RWKV Peng et al. (2023) combines the efficient parallelizable training of transformers with the efficient inference of RNNs while mitigating their limitations. TTT Sun et al. (2024) replaces the hidden state of an RNN with a machine learning model. TTT compresses context through actual gradient descent on input tokens. RetNet Sun et al. (2023) theoretically derives the connection between recurrence and attention, simultaneously achieves training parallelism, low-cost inference, and good performance.

#### B.1.2 SSM

##### Mamba

introduced by Gu and Dao (2023) , is a recent family of autoregressive language models based on state space models (SSMs). Mamba employs a unique architecture called MambaBlock, which replaces the attention and MLP blocks used in Transformer layers.

Specifically, Mamba maps a sequence of tokens x = [ x 1 , x 2 , … , x T ] x=[x_{1},x_{2},\ldots,x_{T}] to a probability distribution over the next token y y . Each token x i x_{i} is first embedded into a hidden state of size d d as h i ( 0 ) h_{i}^{(0)} , which is then transformed sequentially by a series of MambaBlocks. The hidden state h i ( ℓ ) h_{i}^{(\ell)} after the ℓ \ell -th MambaBlock is computed as follows:

h i ( ℓ ) = h i ( ℓ − 1 ) + o i ( ℓ ) \displaystyle h_{i}^{(\ell)}=h_{i}^{(\ell-1)}+o_{i}^{(\ell)} (9)

The output o i ( ℓ ) o_{i}^{(\ell)} of the ℓ \ell -th MambaBlock for the i i -th token is a combination of s i ( ℓ ) s_{i}^{(\ell)} (from Conv and SSM operations) and g i ( ℓ ) g_{i}^{(\ell)} (a gating mechanism):

o i ( ℓ ) \displaystyle o_{i}^{(\ell)} = MambaBlock ( ℓ ) ​ [ h 1 ( ℓ − 1 ) , h 2 ( ℓ − 1 ) , … , h i ( ℓ − 1 ) ] \displaystyle=\text{MambaBlock}^{(\ell)}\left[h_{1}^{(\ell-1)},h_{2}^{(\ell-1)},\ldots,h_{i}^{(\ell-1)}\right] (10) = W o ( ℓ ) ​ [ s i ( ℓ ) ⊗ g i ( ℓ ) ] \displaystyle=W_{o}^{(\ell)}\left[s_{i}^{(\ell)}\otimes g_{i}^{(\ell)}\right]

Here, ⊗ \otimes denotes element-wise multiplication. The calculation of s i ( ℓ ) s_{i}^{(\ell)} is as follows:

a i ( ℓ ) = W a ( ℓ ) ​ h i ( ℓ ) \displaystyle a_{i}^{(\ell)}=W_{a}^{(\ell)}h_{i}^{(\ell)} (11)

c 1 ( ℓ ) , c 2 ( ℓ ) , … , c i ( ℓ ) = \displaystyle c_{1}^{(\ell)},c_{2}^{(\ell)},\ldots,c_{i}^{(\ell)}= (12) SiLU ​ [ Conv1D ​ [ a 1 ( ℓ ) , a 2 ( ℓ ) , … , a i ( ℓ ) ] ] \displaystyle\text{SiLU}\left[\text{Conv1D}\left[a_{1}^{(\ell)},a_{2}^{(\ell)},\ldots,a_{i}^{(\ell)}\right]\right]

s i ( ℓ ) = selective-SSM ​ [ c 1 ( ℓ ) , c 2 ( ℓ ) , … , c i ( ℓ ) ] \displaystyle s_{i}^{(\ell)}=\text{selective-SSM}\left[c_{1}^{(\ell)},c_{2}^{(\ell)},\ldots,c_{i}^{(\ell)}\right] (13)

The operations in Equations ( 12 ) and ( 13 ) correspond to Conv and SSM operations, respectively. The gating mechanism g i ( ℓ ) g_{i}^{(\ell)} is given by:

g i ( ℓ ) = SiLU ​ [ W g ( ℓ ) ​ h i ( ℓ − 1 ) ] \displaystyle g_{i}^{(\ell)}=\text{SiLU}\left[W_{g}^{(\ell)}h_{i}^{(\ell-1)}\right] (14)

The formulas and concepts used here are adapted from Sharma et al. (2024) .

Compared to Transformer, Mamba’s design enables more efficient parallel training and effectively captures dependencies in sequences, making it suitable for various natural language processing tasks.

#### B.1.3 Vision and Multi-modal Models

In the realm of vision and multi-modal models, various architectures have emerged, each with its unique approach to tackling complex visual tasks. For example, GANs (Generative Adversarial Nets) Goodfellow et al. (2014) consist of two neural networks: a generator and a discriminator. Through adversarial learning, the generator aims to produce realistic data samples (such as images), while the discriminator attempts to distinguish between real and generated data. Diffusion Model Li et al. (2023c) ; Sohl-Dickstein et al. (2015) is a powerful tool for generating high-quality images and data. It simulates a diffusion process by gradually adding and removing noise to achieve data generation. ResNet (Residual Network) He et al. (2016) introduced residual learning, revolutionizing deep network training by improving efficiency and performance through skip connections. ViT (Vision Transformer) Dosovitskiy et al. (2021) integrated the Transformer architecture into vision tasks, capturing long-range dependencies by processing image patches.

### B.2 Knowledge Mechanisms in Other Architectures

Surprisingly, similar mechanisms as those found in transformer-based LLMs have also been discovered in other architectural models. Specifically, Mamba employs the knowledge memorization mechanism similar to Transformer Sharma et al. (2024) . Vision and multi-modal architectures also adopt function region ( Modular Region Hypothesis ) for knowledge utilization Pan et al. (2023a) ; Schwettmann et al. (2023a) ; Koh et al. (2020) ; Bau et al. (2017) , e.g., multi-modal neuron regions are responsible for multi-modal tasks. Besides, the connections hypothesis between neurons is found in vision architecture models Olah et al. (2020) . Olah et al. (2020) further suggest that different types of knowledge reuse partial components, e.g., cars and cats reuse the same neurons ( Reuse Hypothesis ). As for the Dynamic Intelligence Hypothesis, it inherently focuses on entire artificial neural models. Generally, neural models across various architectures, trained with different objectives on different data and modalities, are converging to a shared statistical model of reality in their representation spaces Huh et al. (2024) . These neural models may tend to share similar knowledge mechanisms and imagination Zhou et al. (2024)

### B.3 Machine and Human

Analogous to the Hominidae Family in biological taxonomy, artificial neural models can be regarded as Neural Model Family: • Family: Neural Model, likened to “Hominidae”.

• Genus: Transformer architecture, Mamba architecture, etc., likened to “Homo” and “Pan”.

• Species: BERT, GPT, Llama, Mistral, Mamba, etc., likened to “Sapiens”, “Pan troglodytes”, and “Pan paniscus”.

Metaphorically, Llama-7B, Llama-13B, Llama-70B, etc., can be viewed as the infancy, childhood, and adulthood of humans. Shah et al. (2024) further find that, regardless of model size, the developmental trajectories of PLMs consistently exhibit a window of maximal alignment with human cognitive development. Therefore, we hypothesize that artificial neural networks (machine) and biological neural networks (human) tend to converge to universality intelligence. In other words, huamn and machine share similar features and circuits.

Specifically, extensive evidences demonstrate that machine and human share the same mechanism of knowledge memorization, i.e., modular region and connection de Schotten et al. (2022) . The activations of modern language models can also linearly map onto the brain responses to speech Caucheteux et al. (2023) . Caucheteux et al. (2023) pioneer the explanation via predictive coding theory: while transformer-based LLMs are optimized to predict nearby words, the human brain would continuously predict a hierarchy of representations that spans multiple timescales. The above phenomenon indicates that machine and human share similar underlying mechanisms of knowledge Sucholutsky et al. (2023) ; Chan et al. (2023) ; Kornblith et al. (2019) , irrespective of their specific configurations, process and comprehend information. This could be due to inbuilt inductive biases Sæbø and Brovold (2024) in neural networks or natural abstractions Chan et al. (2023) – concepts favored by the natural world that any cognitive system would naturally gravitate towards Bereska and Gavves (2024) .

## Appendix C Tools for Mechanism Analysis

Numerous tools exist for interpreting knowledge mechanisms in LLMs. TransformerLens Nanda and Bloom (2022) is a library for the mechanistic interpretability using observation and intervention. TransformerLens allows users to cache, remove, or replace internal activations during model running. XMD Lee et al. (2023a) provides various forms of feedback via an intuitiveness, which enable explanations align with the user feedback. NeuroX Dalvi et al. (2023) implements various interpretation methods under a unified API then provides interpretability of LLMs. PatchScope Ghandeharioun et al. (2024) is a tool developed by Google that employs a novel model to elucidate the hidden states in the original model. Transformer Debugger Mossing et al. (2024) , an interpretability tool from OpenAI, utilizes GPT-4 and sparse auto-encoders to explain language neurons. Sparse autoencoders Gao et al. (2024b) leverages sparse auto-encoders to extract interpretable features from a language model by reconstructing activations from a sparse bottleneck layer. Transcoders Dunefsky et al. (2024) decomposes model computations involving MLPs into interpretable circuits.

## Instructions for reporting errors

We are continuing to improve HTML versions of papers, and your feedback helps enhance accessibility and mobile support. To report errors in the HTML that will help us improve conversion and rendering, choose any of the methods listed below:

Click the "Report Issue" ( ) button, located in the page header.

Tip: You can select the relevant text first, to include it in your report.

Our team has already identified the following issues . We appreciate your time reviewing and reporting rendering errors we may not have found yet. Your efforts will help us improve the HTML versions for all readers, because disability should not be a barrier to accessing research. Thank you for your continued support in championing open access for all.

Have a free development cycle? Help support accessibility at arXiv! Our collaborators at LaTeXML maintain a list of packages that need conversion , and welcome developer contributions .