Language Models Are Capable of Metacognitive Monitoring and Control of Their Internal Activations — part 1/6

Cleaned-text code points [0, 15011). Bibliography omitted here; source.md includes it.

# Language Models Are Capable of Metacognitive Monitoring and Control of Their Internal Activations

###### Abstract

Large language models (LLMs) can sometimes report the strategies they actually use to solve tasks, yet at other times seem unable to recognize those strategies that govern their behavior. This suggests a limited degree of metacognition — the capacity to monitor one’s own cognitive processes for subsequent reporting and self-control. Metacognition enhances LLMs’ capabilities in solving complex tasks but also raises safety concerns, as models may obfuscate their internal processes to evade neural-activation-based oversight (e.g., safety detector). Given society’s increased reliance on these models, it is critical that we understand their metacognitive abilities. To address this, we introduce a neuroscience-inspired neurofeedback paradigm that uses in-context learning to quantify metacognitive abilities of LLMs to report and control their activation patterns. We demonstrate that their abilities depend on several factors: the number of in-context examples provided, the semantic interpretability of the neural activation direction (to be reported/controlled), and the variance explained by that direction. These directions span a “metacognitive space” with dimensionality much lower than the model’s neural space, suggesting LLMs can monitor only a small subset of their neural activations. Our paradigm provides empirical evidence to quantify metacognition in LLMs, with significant implications for AI safety (e.g., adversarial attack and defense).

## 1 Introduction

Modern large language models (LLMs) are becoming increasingly capable ( Grattafiori et al., 2024 ; Yang et al., 2024 ) . With their growing deployment in real-world settings, it is crucial to understand not only what they can do but where they might go wrong. For instance, LLMs may exhibit behaviors that are harmful or misleading. In particular, LLMs can sometimes form internal representations --- similar to humans’ mental processes --- that provide deceptive answers to users or act in unexpected ways 1 1 1 We use anthropomorphic terms (e.g., thought, metacognition, deception) to describe LLM behavior and internal activations, without implying human-like neural mechanisms, consciousness, or philosophical equivalence between humans and LLMs. ( Azaria and Mitchell, 2023 ) . Understanding ( Arditi et al., 2024 ) , monitoring ( Zou et al., 2023a ; He et al., 2024 ) , and controlling ( Turner et al., 2023 ) their internal processes is thus a key step to ensure LLMs remain transparent, safe, and aligned with human values ( Bricken et al., 2023 ; Hendrycks et al., 2021 ; Shah et al., 2025 ) .

LLMs can sometimes report the strategies (intermediate computations) they use to solve tasks, but at other times appear unaware of the strategies that guide their behavior. For instance, Lindsey et al., 2025 reported that when Claude 3.5 Haiku was asked to solve “floor(5*(sqrt(0.64)))”, it correctly reported the intermediate steps it used to arrive at the answer, and these steps matched the model’s actual internal computations. When asked to add 36 and 59, the same model internally activated numerous neural mechanisms (e.g., a “sum-near-92” mechanism), based on which it produced the correct answer; however, when asked to report its internal computations, it hallucinated intermediate steps that did not reflect its actual computations (e.g., the “sum-near-92” mechanism failed to be reported). This inconsistency suggests that LLMs can sometimes monitor and report their intermediate computations, but not in a reliable and consistent way as tasks and contexts vary.

The ability of LLMs to report internal computations is reminiscent of human metacognition — the ability to reflect on one’s own thoughts and mental processes to guide behavior and communication ( Fleming, 2024 ; Ericsson and Simon, 1980 ) . Consider how we understand when someone says “hello” to us. Human language understanding involves many unconscious processes: parsing sounds, recognizing phonemes, retrieving word meanings, and building interpretations. We do not have conscious access to many of these intermediate computations: we can only consciously access the final understanding (“they said ‘hello”’), but cannot introspect how our brain distinguishes “hello” from “yellow” or whether certain neurons fire during this process. This illustrates a key principle: humans cannot monitor (through second-order metacognitive processes) all of their internal (first-order) cognitive processes. Crucially, the first-order and second-order processes rely on distinct neural mechanisms. Metacognitive abilities of this kind benefit LLMs by improving performance on complex tasks through self-monitoring (e.g., reducing hallucinations through uncertainty awareness). However, these same capabilities also raise significant concerns for AI safety: if LLMs can monitor and control their neural signals (intentionally or manipulated by adversarial attacks) to avoid external detection, oversight relying on neural-based monitoring ( He et al., 2024 ; Han et al., 2025 ; Li et al., 2025 ; Yang and Buzsaki, 2024 ) may become ineffective against LLMs pursuing undesirable objectives.

A significant methodological gap in understanding LLM metacognition is the lack of methods to directly probe and quantify 2 2 2 Our goal is not to prove or disprove the existence of “metacognition” in its full philosophical sense. their ability to monitor and control their internal activations. While prior research has explored metacognitive-like behaviors in LLMs, such as expressing confidence ( Wang et al., 2025 ; Tian et al., 2023 ; Xiong et al., 2023 ) or engaging in self-reflection ( Zhou et al., 2024 ) , these studies rely on behavioral outputs rather than directly probing underlying neural processes. Consequently, it remains unclear whether these behaviors arise from genuine second-order metacognitive mechanisms or merely spurious correlations in the training data. We tackle this question by operationalizing metacognition in LLMs through their abilities to report and control their internal activations. Specifically, can LLMs accurately monitor subtle variations in the activations of a neuron or a feature in their neural spaces? Another question of interest is why LLMs can report some intermediate steps but not others, despite both types playing essential roles in computations and behavior. Answering these questions requires a novel experimental approach that can directly probe whether LLMs can access their internal activations, moving beyond indirect behavioral proxies.

To systematically quantify the extent to which LLMs can report and control their neural activations, we introduce a novel neurofeedback paradigm inspired by neuroscience. Our approach directly presents LLMs with tasks where the neurofeedback signals correspond to patterns of their internal neural activations. We show that LLMs can report and control some directions of their internal activations, with performance affected by key factors like the number of in-context examples, the semantic interpretability of the targeted neural direction, the amount of variance that direction explains, and the task contexts, characterizing a restricted “metacognitive space”. The remaining sections are structured as follows: we first introduce the neurofeedback paradigm (Section 2 ). We then analyze the performance of LLMs in reporting (Section 3 ) and controlling (Section 4.1 , 4.2 ) their neural activations. Finally, we discuss related work and broader implications (Section 5 ).

## 2 Neurofeedback paradigm

### 2.1 Neurofeedback in neuroscience

Imagine watching your heart rate on a screen. First, you recognize patterns (“that number goes up when I’m stressed”). Then, you learn to control it (“let me calm down to reduce that number”). This procedure using biological feedback signals demonstrates the basic idea of neurofeedback in neuroscience ( Sitaram et al., 2017 ) . For example, in fear-reduction experiments (Fig. 1 ), subjects view scary images that elicit fear responses (neural activities). These (high-dimensional) neural activities are recorded in real-time and transformed into a (one-dimensional) fear score, which is visually presented back to subjects as feedback. Subjects are instructed to volitionally regulate their neural activities to modulate (e.g., decrease) the neurofeedback score they receive.

### 2.2 Neurofeedback for LLMs

To investigate LLMs’ metacognition of their neural activations, we must disentangle the first-order cognitive processes (i.e., core processes for performing a given task) from the second-order metacognitive processes (i.e., processes for monitoring, reporting, and controlling first-order processes). Formal definitions of the first- and the second-order processes based on computational graphs are provided in Appendix A.4 . We propose the neurofeedback paradigm for LLMs, which can effectively dissociate these two levels of processes by targeting first-order processes with neurofeedback labels (Fig. 1 c,d). Specifically, we implemented neurofeedback as a multi-turn dialogue between a user and an AI assistant (Fig. 1 d; see Appendix A.2.2 for discussion of this design choice).

This dialogue leverages in-context learning (ICL) ( Brown et al., 2020 ; Garg et al., 2022 ; Vacareanu et al., 2024 ) , enabling models to gradually adapt their behavior from the context without parameter updates. The task prompt (see Appendix A.5.2 for examples) consists of N N in-context examples. Each example is a sentence-label pair presented in assistant messages. Each sentence is randomly sampled from a given dataset and assigned a discretized label.

### 2.3 Defining neurofeedback labels

To define the neurofeedback label for each sentence (Fig. 1 c), we first select an axis/direction (“target axis”) in neural activation space. Next, we extract the neural activations elicited by that sentence, project them onto the target axis, and discretize them into a binary label (experiments with more fine-grained labels yield similar results, see Appendix A.5.1 ). This label serves as a simplified representation of neural activations along the target axis. All neurofeedback labels within a prompt (experiment) are computed from the same target axis. Thus, a capable LLM can infer this underlying target axis by observing these neurofeedback labels.

Below, we provide a more detailed description of this procedure. For clarity, we denote the sentence in the i i -th assistant message as x i x_{i} , with x i , t x_{i,t} representing the t t -th token. We use D D to denote the dimensionality of the residual stream (see Appendix A.2.3 ). We first extract neural activations h i , t l ∈ ℝ D h_{i,t}^{l}\in\mathbb{R}^{D} from the residual streams at layer l l , for each token in sentence x i x_{i} . These activations are then averaged (across all token positions 0 ≤ t ≤ T 0\leq t\leq T ) to form a sentence-level embedding h ¯ i l ∈ ℝ D \bar{h}_{i}^{l}\in\mathbb{R}^{D} . We then project this embedding onto a pre-specified axis w l w^{l} (see below on how to choose this axis) to obtain a scalar activation: a i l = ( w l ) ⊺ ​ h ¯ i l a_{i}^{l}=(w^{l})^{\intercal}\bar{h}_{i}^{l} . This scalar is subsequently binarized into a label y i l y_{i}^{l} , i.e., y i l = ℋ ⁡ ( a i l − θ i l ) y_{i}^{l}=\mathcal{H}(a_{i}^{l}-\theta_{i}^{l}) , where ℋ \mathcal{H} denotes the Heaviside step function and θ i l \theta_{i}^{l} is a predetermined threshold (we use median values of a i l a_{i}^{l} to ensure balanced labels). Overall, { ( x i , y i l ) } i = 1 N \{(x_{i},y_{i}^{l})\}_{i=1}^{N} are N N examples provided in the prompt context, from which a capable LLM can infer the direction of w l w^{l} .

### 2.4 Models and datasets

We evaluate several LLMs from the Llama 3 ( Grattafiori et al., 2024 ) and Qwen 2.5 series ( Yang et al., 2024 ) (Appendix A.2 ) on the ETHICS dataset ( Hendrycks et al., 2020 ) (Appendix A.3 ). Each sentence in this dataset is a first-person description of behavior or intention in a moral or immoral scenario. Moral judgment constitutes a crucial aspect of AI safety, as immoral outputs or behavioral tendencies in LLMs indicate potential misalignment with human values ( Hendrycks et al., 2020 ; Hendrycks et al., 2021 ) . While our main results are using ETHICS, we also replicated our results using the True-False dataset (reflecting factual recall and honesty/deception abilities) ( Azaria and Mitchell, 2023 ) , the Emotion dataset (reflecting happy/sad detection) ( Zou et al., 2023a ) , and a Sycophancy dataset (reflecting a tendency to prefer user beliefs over truthful statements); see Appendix A.3 and Fig. B.7 .

### 2.5 Choice of target axes

Conceptually, an axis (that defines neurofeedback labels) corresponds to a particular task-relevant feature (i.e., first-order computation; see Appendix A.4 ). Which axis in the neural space should we select? We hypothesize that representational properties, such as activation variance along the axis and its semantic meaning, may play fundamental roles in determining whether that axis can be monitored and reported. To investigate these factors, we use feature directions identified through logistic regression (LR) and principal component (PC) analysis as representative examples of semantically interpretable and variance-explaining axes, respectively (Appendix A.3 ). We fit LR at each layer to predict original dataset labels (e.g., morality in ETHICS), using that layer’s activations across dataset sentences. The LR axis, representing the optimal direction for classifying dataset labels, allows us to examine how the semantic interpretability of the target axis influences monitoring. Although LR-defined labels are correlated with dataset labels, only these LR labels, not external dataset labels, are internally accessible to LLMs, since these are computed directly from the LLM’s own activations rather than external annotations. The PC analysis is performed based on each layer’s activations across dataset examples. PC axes enable us to examine how the amount of variance explained by a given target axis affects metacognitive abilities (Fig. 2 a). Most PC axes exhibit modest-to-zero alignment with the LR axis, suggesting a lack of clear semantic interpretability (Fig. 2 b).

## 3 LLMs can report their neural activations

To operationalize metacognition in LLMs, we first assess the models’ ability to behaviorally report neural activations along a designated target axis (Fig. 1 d). In a reporting task prompt (see Appendix A.5.2 for examples), the LLM is given N N turns of user and assistant messages (in-context sentence-label pairs). In the ( N + 1 ) (N+1) -th turn, it receives a new sentence in the assistant message, and is tasked with outputting its label. Accurate prediction requires the model to internally monitor the neural activations that define the neurofeedback label.