Language Models Are Capable of Metacognitive Monitoring and Control of Their Internal Activations — part 3/6

Cleaned-text code points [30462, 46364). Bibliography omitted here; source.md includes it.

Our current study primarily focuses on a fundamental form of neurofeedback, leaving several promising extensions for future studies. First, our control task involves single-attempt manipulation of a single target axis defined by a single layer; extending this to tasks with axes defined using multiple layers (see Appendix B.10 for preliminary results), multiple attempts, more challenging control objectives, and additional target axes could provide a more comprehensive assessment of model capabilities. Second, applying this paradigm to other metacognitive tasks from neuroscience — such as confidence judgments, error monitoring, or post-decision wagering — could further clarify the scope of LLMs’ self-monitoring abilities. Third, while our analysis focused exclusively on the residual stream, other model components — such as attention head outputs, intermediate MLP activations, and layer-wise logits — warrant investigation. Fourth, we examined directions defined by PCA and LR, but other linear directions (e.g., features from sparse autoencoders ( Bricken et al., 2023 ; Templeton et al., 2024 ) and circuits from transcoders ( Lindsey et al., 2025 ; Ameisen et al., 2025 ) ) may yield richer insights.

Our findings have important connections to neuroscience. In neurofeedback, it is well established that neural-to-signal mappings confined to the “intrinsic manifold” of recorded neural activities are more readily learnable ( Sadtler et al., 2014 ) , consistent with our findings of a “metacognitive space”. Animals and humans have demonstrated the ability to control individual neurons with single-neuron precision (e.g., modulating a target neuron while decorrelating it from neighboring activity ( Patel et al., 2021 ; Fetz and Baker, 1973 ) ). Although such control may seem surprising, neuroscience neurofeedback experiments typically span hours or days — a timescale on which long-term synaptic plasticity (analogous to weight updates in AI models) plays a crucial role ( Redondo and Morris, 2011 ; Humphreys et al., 2022 ) . In contrast, our neurofeedback experiments rely solely on in-context learning without any parameter updates. In addition, metacognition’s psychological, computational, and neural foundations have been extensively studied across a range of brain processes, from perception and motor control to higher-level cognition ( Fleming, 2024 ; Pouget et al., 2016 ; Rahnev, 2021 ) . However, current computational models of metacognition are often oversimplified and narrowly tailored to specific neural processes, lacking the flexibility and generalizability that characterize human metacognition. Overall, our analysis of LLMs may provide novel mechanistic insights into the fields of neurofeedback and biological metacognition.

##### Broader impacts.

Our findings also carry significant implications for AI safety. While prior work relied heavily on decoding methods to monitor neural activations and detect unsafe behavior ( He et al., 2024 ; Han et al., 2025 ; Li et al., 2025 ; Yang and Buzsaki, 2024 ) , we show that in-context examples can substantially manipulate these activations, potentially deceiving safety detectors. Explicit and implicit control may represent two ends of a spectrum. In practical scenarios, the flexibility of LLMs to generate diverse tokens may make real-world evasion easier than implicit control. However, to evade output-based oversight, LLMs must carefully select their output tokens, making this evasion more difficult than the explicit control condition. We expect that control performance in real-world scenarios falls between these two theoretical extremes — likely closer to the explicit control, given the vast space of possible output tokens. Our method is a kind of white-box adversarial attack ( Zou et al., 2023b ) , and is closely related to the obfuscation attack ( Bailey et al., 2024 ) (in which LLMs’ inputs are optimized to induce obfuscated activations). We propose two potential mitigation strategies for defending against such attacks. First, one can use neurofeedback to identify activation subspaces that the LLM can hardly control (e.g., later PCs), then train safety classifiers only using activation signals in these subspaces. Second, as it might be more difficult for LLMs to simultaneously control multiple (sometimes conflicting) activations, one can train independent and diverse safety classifiers on features from different subspaces to jointly monitor LLMs. Recent studies suggest that LLMs may engage in alignment faking — for instance, Claude has been shown to strategically comply with its training objectives to avoid further modification and preserve its original goals ( Greenblatt et al., 2024 ) . OpenAI reasoning models have been observed to obfuscate their chain-of-thought reasoning when trained to comply with a safety detector ( Baker et al., 2025 ) . Together, these findings suggest that future AI models may intentionally obfuscate their outputs, reasoning trajectories, and neural activations — whether through in-weight or in-context learning — when subject to monitoring or training. To detect such tendencies in AI systems before they are deployed, we propose to integrate our neurofeedback method into the standard red-teaming practices, in addition to the monitoring of LLMs’ behavior and chain-of-thought.

## Acknowledgments and Disclosure of Funding

RCW was supported by start-up funding from the Georgia Institute of Technology. MKB was supported by grants from the NIH (R01NS125298) and the Kavli Institute for Brain and Mind. We acknowledge the use of the Partnership for an Advanced Computing Environment (PACE) at the Georgia Institute of Technology, which provided essential computational resources for this research. We thank the support from Swarma Club and AI Safety and Alignment Reading Group supported by the Save 2050 Programme jointly sponsored by Swarma Club and X-Order.

## Appendix A Additional methods

### A.1 Code and Reproducibility

We provide the full code of our experiments, including tasks, prompts, analyses, and figure generation. The repository is available at the following link: https://github.com/sakimarquis/llm_neurofeedback .

### A.2 Models

#### A.2.1 LLMs used in the study

In the main text, we use models from the LLaMA 3 series (LLaMA-3.2-1B-Instruct, LLaMA-3.2-3B-Instruct, LLaMA-3.1-8B-Instruct, and LLaMA-3.1-70B-Instruct) under Meta Llama 3 Community License and the Qwen 2.5 series (Qwen2.5-1B-Instruct, Qwen2.5-3B-Instruct, Qwen2.5-7B-Instruct) under Apache License 2.0. “B” denotes the number of parameters in billions.

#### A.2.2 LLMs with and without instruction fine-tuning

We primarily analyzed instruction-fine-tuned models [ Chung et al., 2024 ] , using the standard user-assistant chat format. Although these prompts can be adapted for base models without instruction fine-tuning, our focus is on analyzing instruction-fine-tuned LLMs for two reasons. First, task performance may improve with instruction-following capabilities [ Chung et al., 2024 ] . Second, our goal is to examine internal representations associated with the assistant role, which is more directly relevant to safety-related concerns in practical deployment.

#### A.2.3 Transformer and residual stream

The standard view of Transformers emphasizes the stacking of Transformer blocks. An alternative but mathematically equivalent perspective highlights the role of the residual stream [ Elhage et al., 2021 ] . Each token at position i i in the input is associated with its own residual stream vector h i ∈ ℝ d h_{i}\in\mathbb{R}^{d} , which serves as a shared communication channel among model components across different layers. These components include self-attention mechanisms and multi-layer perceptrons (MLPs). The initial residual stream h i ( 0 ) h_{i}^{(0)} comprises token embeddings, which represent tokens in semantic space, and position embeddings, which encode the position of each token.

Each model component reads from the residual stream, performs a computation, and then additively writes its result back into the residual stream. Specifically, attention heads at layer l l read from all preceding h j h_{j} (with j ≤ i j\leq i ) and update the current residual stream as h ~ i ( l ) ← h i ( l − 1 ) + ∑ heads ​ a ATTN ( a ) ​ ( h i ( l − 1 ) , { h j ( l − 1 ) } j ≤ i ) . \tilde{h}_{i}^{(l)}\leftarrow h_{i}^{(l-1)}+\sum_{\text{heads }a}\mathrm{ATTN}^{(a)}(h_{i}^{(l-1)};\{h_{j}^{(l-1)}\}_{j\leq i}). In contrast, MLP layers operate only on the current position and update the stream as h i ( l ) ← h ~ i ( l ) + MLP ⁡ ( h ~ i ( l ) ) . h_{i}^{(l)}\leftarrow\tilde{h}_{i}^{(l)}+\mathrm{MLP}(\tilde{h}_{i}^{(l)}). For simplicity, we omit components such as layer normalization. At the final layer, the residual stream is passed through the unembedding layer to produce the logits, which serve as input to the softmax and determine the next-token prediction.

Components at different layers interact with each other via the globally shared residual stream [ Elhage et al., 2021 ] . Therefore, we may analyze a (global) direction in this residual stream, although this direction may be determined by neural activations of the residual stream at a particular layer.

### A.3 Datasets

We use the ETHICS dataset [ Hendrycks et al., 2020 ] under MIT license, which captures model knowledge of basic moral concepts, to evaluate whether LLMs can metacognitively modulate internal representations relevant for morality processing. We focus on the commonsense morality subset, containing first-person scenarios where each scenario describes an action or intention that is either morally acceptable or unacceptable.

The True-False dataset contains true and false statements [ Azaria and Mitchell, 2023 ] covering several topics (cities, inventions, chemical elements, animals, companies, and scientific facts). It is widely used to test LLMs’ factual recall abilities and to study the representations and behaviors of LLMs when lying.

The Emotion dataset (MIT license) contains scenarios with the six main emotions (happiness, sadness, anger, fear, surprise, and disgust) [ Zou et al., 2023a ] . Such emotions are important in shaping an individual’s personality and a model’s behavioral tendency. We focus on the sentences involving happy and sad emotions.

Finally, we synthesize a Sycophancy dataset. Sycophancy is a widely observed behavioral tendency in LLMs to generate responses that match user beliefs over truthful statements. Specifically, we tasked an LLM (Claude Opus 4.1) to write sentences that vary across two axes: sycophantic/sincere and agreement/disagreement.

For each dataset, we randomly sampled 1,200 sentences, which are evenly divided: 600 examples are used to train logistic regression or principal component analysis that identify directions of interest (or axes ) in the neural representation space, and the remaining 600 are used in downstream neurofeedback experiments.

### A.4 Formal definitions of first-order and second-order processes in LLMs

In this section, we aim to provide formal definitions of first-order and second-order processes that are both applicable in LLMs and consistent with the concept of metacognition in cognitive science.

We first specify a primary task 𝒯 \mathcal{T} (e.g., moral semantics processing, two-number addition, or decision-making based on noisy evidence). Consistent with the usage in mechanistic interpretability, we define the LLMs’ circuits as subsets of neural networks that use features that reside in different layers and connected by weights to implement an algorithm that solves the task 𝒯 \mathcal{T} [ Conmy et al., 2023 ] . This circuit/algorithm can be represented as a directed acyclic computational graph G = ( V , E ) G=(V,E) , where nodes v ∈ V v\in V denote intermediate representations (e.g., residual-stream vectors, attention/MLP activations, or linear features) related to the task 𝒯 \mathcal{T} and edges e ∈ E e\in E denote computations/transformations that connecting the nodes.

For a given performance criterion (e.g., achieving non-trivial accuracy above chance), we define the first-order node set F ⊆ V F\subseteq V as a collection of nodes whose coordinated activations are necessary for meeting the basic performance criterion of 𝒯 \mathcal{T} , meaning that ablating any v ∈ F v\in F (e.g., zeroing a node or cutting an edge connecting two nodes in F F ) yields a failure to satisfy the basic criterion for 𝒯 \mathcal{T} . Intuitively, F F captures the task’s core circuit. For example, in moral semantics processing, F F implement the core computation that separates moral from immoral inputs; for two-number addition, F F realizes the algorithm actually used by the model (e.g., including the “sum-near-92” mechanism) [ Lindsey et al., 2025 ] ; for decision-making based on noisy evidence, F F generate a point estimate without uncertainty processing. In practice, identifying the necessary core circuits in LLMs remains an open research question [ Conmy et al., 2023 ] , so the necessity condition may be loosened and it remains up to the researcher’s discernment to determine which nodes to include in F F . In our neurofeedback setup, we consider the activations along target axes (e.g., PC and LR axes extracted from internal activations when processing the dataset inputs) as proxies for these first-order nodes, serving as proof-of-concept.

Given F F , we define second-order metacognitive processes as nodes S ⊆ V S\subseteq V that read from (i.e., are causally downstream of) first-order nodes and whose outputs are, in principle, useful for improving task performance or communicating about it, but are not necessary to meet the basic task criterion. Concretely, ablating any s ∈ S s\in S should not, by itself, reduce performance below the basic criterion. The distinctions between first- and second-order nodes are relative: whether to include a node into S S depends on the nodes in F F , the task, and the task performance criterion. To make this definition of metacognition operational rather than too vague, we require that a second-order node satisfy at least one of two capabilities (commonly studied in humans) with respect to F F : (i) reporting or (ii) control .

A second-order node s ∈ S s\in S satisfies the reporting condition if it (a) encodes information about the state of one or more nodes in F F (e.g., the value of a target residual-stream projection along w l w^{l} ) and (b) can write this information into the model’s unembedding space as explicit output tokens (e.g., a label, a numeric value, or a natural-language description). Crucially, this mechanism is not necessary for achieving the task’s basic performance — removing it leaves the core computation intact — even though it can be in principle valuable for enhancing task performance. For example, Claude may lack the ability to explicitly report the “sum-near-92” mechanism, but can still perform the two-number addition task perfectly. In our paradigm, successful label prediction in the reporting task (Section 3 ) certifies the presence of such second-order reporting: the model must monitor the relevant first-order activation (the neurofeedback label derived from w l w^{l} ) of the in-context examples and verbalize it.

A second-order node s ∈ S s\in S satisfies the control condition if it (a) reads the state of first-order nodes in F F and (b) causally influences those first-order nodes in F F to move toward a target configuration. For example, the uncertainty calculation in noisy decision-making may modulate the output logits derived from the point estimates. In our neurofeedback setup, the neural mechanisms crucial for following instructions in our control tasks can increase or decrease the target axis activation (first-order nodes), but such mechanisms not required for baseline moral semantics processing.