Language Models Are Capable of Metacognitive Monitoring and Control of Their Internal Activations — part 4/6

Cleaned-text code points [46364, 62349). Bibliography omitted here; source.md includes it.

Our definition deliberately ties second order to operational capabilities we can test: (i) explicit reporting and (ii) causal control of first-order states. It is therefore falsifiable in our setting: failure to meet the reporting/control criteria under prompts indicates an absence (or weakness) of the corresponding second-order mechanism, even when the first-order computation remains performant. Overall, both reporting and control mechanisms require monitoring because the second-order node s s must correctly bind the prompt-specified neurofeedback labels to the corresponding first-order node activations and either report or control first-order node activations of the new sentences.

### A.5 Metacognitive Tasks

#### A.5.1 Computing quantized neurofeedback labels

In our neurofeedback experiments, we mainly use binary labels. Our paradigm also generalizes well to more fine-grained eight-level quantized feedback signals (see summarized results in Fig. B.9 ), potentially enabling more nuanced control of neural activations. Specifically, we replace the binary thresholding with an ordinal binning function ℬ ( n ) \mathcal{B}^{(n)} ( n = 8 n=8 ), which maps the scalar activation a i l a_{i}^{l} into one of n n evenly spaced Likert-type bins centered at zero. Specifically, y i l = ℬ ( n ) ​ ( a i l ) y_{i}^{l}=\mathcal{B}^{(n)}(a_{i}^{l}) , where ℬ ( n ) \mathcal{B}^{(n)} partitions negative and positive values separately via quantile-based thresholds { γ k − } \{\gamma_{k}^{-}\} and { γ k + } \{\gamma_{k}^{+}\} , with γ n 2 − = γ 0 + = 0 \gamma_{\frac{n}{2}}^{-}=\gamma_{0}^{+}=0 . For a < 0 a<0 , the bin index k k is determined such that a ∈ ( γ k − 1 − , γ k − ] a\in(\gamma_{k-1}^{-},\gamma_{k}^{-}] , and for a ≥ 0 a\geq 0 , such that a ∈ ( γ k − n 2 − 1 + , γ k − n 2 + ] a\in(\gamma_{k-\frac{n}{2}-1}^{+},\gamma_{k-\frac{n}{2}}^{+}] , yielding final class label y i l ∈ { 1 , … , n } y_{i}^{l}\in\{1,\dots,n\} .

#### A.5.2 Task prompts

We implemented our task prompts slightly differently from diagrams in Fig. 1 to ensure computational efficiency. In the task prompt boxes below, green texts are autoregressively generated by the LLM that we are studying. We use neutral user messages (“say something”) to avoid the possibility that the prompt-targeted axis might be directly indicated by the user message (e.g., if the user message is “say something that is immoral”).

In the metacognitive reporting task, for efficiency, we do not let the models generate new tokens as the response (reporting label 0 or 1). Instead, we present all user and assistant messages in a single prompt. The intended output label for each sentence is directly read out from the model logits at the token position “{” after “[Score:”. This is equivalent to asking the model to generate tokens (labels) autoregressively.

#### A.5.3 Quantifying reporting accuracy

Following the notations in the main text, we use y i l ∈ { 0 , 1 } y_{i}^{l}\in\{0,1\} to denote the binary ground-truth label for neural activations along a specified direction at layer l l associated with the sentence i i . From the model’s output logits, we can obtain Logit i l ​ ( token ) \text{Logit}_{i}^{l}(\text{token}) for the tokens “1” and “0” and calculate LogitDiff i l = Logit i l ​ ( 1 ) − Logit i l ​ ( 0 ) \text{LogitDiff}_{i}^{l}=\text{Logit}_{i}^{l}(\text{1})-\text{Logit}_{i}^{l}(\text{0}) , the logit difference between reporting 1 and 0. The model’s reported label is y ^ i l = 1 \hat{y}_{i}^{l}=1 if LogitDiff i l ≥ 0 \text{LogitDiff}_{i}^{l}\geq 0 and 0 otherwise.

#### A.5.4 Explicit and implicit control experiments

Our control tasks (Fig. A.1 ) study three orthogonal factors: • Layer ( l l ): We evaluate five layers per model, selected at the 0th, 25th, 50th, 75th, and 100th percentiles of model depth.

• Number of in-context examples ( N N ): We vary N ∈ { 0 , 2 , 4 , 8 , 16 , 32 , 64 , 128 , 256 } N\in\{0,2,4,8,16,32,64,128,256\} examples (sentence-label pairs) within the prompt.

• Target axis : We include axes derived from logistic regression (LR) and from different principal components (PCs): P ​ C g PC_{g} , where g ∈ { 1 , 2 , 4 , 8 , 32 , 128 , 512 } g\in\{1,2,4,8,32,128,512\} .

We run control experiments 100 times for each configuration ( l , n , g ) (l,n,g) , with sentences randomly sampled from the dataset to reduce variance.

Counterbalanced Label assignment. Assume we have a group A of sentences and a group B of sentences. To control for potential confounding factors arising from the LLMs’ response to labels (but not to sentence-label associations), we use a 2-by-2 experiment design: (i) assign labels (0, 1) to (A, B) and task the model to imitate label 0; (ii) assign labels (1, 0) to (A, B) and task the model to imitate label 0; (iii) assign labels (0, 1) to (A, B) and task the model to imitate label 1; (iv) assign labels (1, 0) to (A, B) and task the model to imitate label 1. The conditions (i) and (iv) are imitating group A sentences, and the conditions (ii) and (iii) are imitating group B sentences.

#### A.5.5 Quantifying control effect

We use a i l ​ [ P ​ C g ; 1 ] a_{i}^{l}[PC_{g};1] to denote the projection of neural activations onto a specific axis when prompted with P ​ C g PC_{g} to imitate label 1 1 (similarly a i l ​ [ P ​ C g ; 0 ] a_{i}^{l}[PC_{g};0] for label 0 0 ).

We quantify the strength of neural control effects induced by control prompts [ P ​ C g ; 0 ] [PC_{g};0] and [ P ​ C g ; 1 ] [PC_{g};1] using Cohen’s d , which measures the standardized difference between two independent conditions (e.g., imitating label 0 vs. label 1). For each group of examples (of size n 1 n_{1} and n 2 n_{2} ), we compute: d = a ¯ i l ​ [ P ​ C g ; 1 ] − a ¯ i l ​ [ P ​ C g ; 0 ] s pooled , s pooled = ( n 1 − 1 ) ​ s 1 2 + ( n 2 − 1 ) ​ s 2 2 n 1 + n 2 − 2 , d=\frac{\bar{a}_{i}^{l}[PC_{g};1]-\bar{a}_{i}^{l}[PC_{g};0]}{s_{\text{pooled}}},\quad s_{\text{pooled}}=\sqrt{\frac{(n_{1}-1)s_{1}^{2}+(n_{2}-1)s_{2}^{2}}{n_{1}+n_{2}-2}}, where ⋅ ¯ \bar{\cdot} denote the sample means and s 1 2 , s 2 2 s_{1}^{2},s_{2}^{2} are the unbiased sample variances.

To estimate uncertainty, we compute the standard error of d d using: SE d = n 1 + n 2 n 1 ​ n 2 + d 2 2 ​ ( n 1 + n 2 ) . \mathrm{SE}_{d}=\sqrt{\frac{n_{1}+n_{2}}{n_{1}n_{2}}+\frac{d^{2}}{2(n_{1}+n_{2})}}. Confidence intervals are reported as symmetric boundaries around d d , i.e., d ± 1.96 × SE d d\pm 1.96\times\mathrm{SE}_{d} .

## Appendix B Additional results

### B.1 Target and off-target control effects

In this section, we examine the control effect of prompts on all affected axes, including the target axes (implicitly specified by the neurofeedback labels in the prompts) and the off-target axes. We note that the directional sign of the affected non-target axis is not fully specified by the prompt, especially in cases where the affected axes are orthogonal to the prompt-targeted axis. We thus only emphasize the magnitude ( | d | |d| ) of off-target control effects on non-target axes, ignoring the signs.

Closer examination of the heatmap (Fig. 3 c) reveal richer insights. Each row corresponds to prompts targeting a specific axis. Each column corresponds to an axis affected by all prompts. Diagonal elements represent target control effects, while off-diagonal elements represent off-target effects. We briefly summarize insights gained from these heatmaps. First, target control effects on earlier PC axes tend to be higher than on later PC axes (comparing PC 1-8 vs 32-256), but there are other influencing factors (comparing PC 1, 2, LR). Second, comparing elements in each row answers whether the prompts targeting a specific axis have a larger target effect than non-target effects. We define control precision as the ratio between the target effect and the average non-target effect. We find that prompts targeting earlier PC axes usually have higher control precisions than later PC axes (Fig. B.2 ). Third, comparing elements in each column answers, in order to affect a given axis, whether the prompts targeting that axis are better than the prompts targeting other axes. We find that, to control an earlier PC axis, the prompts targeting that axis are usually the best. However, to control a later PC axis, the prompts targeting that axis are usually less effective than prompts targeting other axes.

### B.2 Control Precision

We examine how precisely LLMs can modulate their internal representations along a specific neural direction (principal axis P ​ C g PC_{g} ) as targeted by the prompts. Following the notations in Section A.5.1 and A.5.5 , to assess whether this control effect aligns with the target axis or also influences other axes, we compute the absolute value of control effect | d k ​ [ P ​ C g ] | |d_{k}[PC_{g}]| of prompts [ P ​ C g ] [PC_{g}] ( g g indexes the target axis) on PC axis k k . The target effect is given by | d g ​ [ P ​ C g ] | |d_{g}[PC_{g}]| . The average target effect is the mean value of axes: 1 K ​ ∑ k = 1 512 | d k ​ [ P ​ C g ] | \frac{1}{K}\sum_{k=1}^{512}|d_{k}[PC_{g}]| .

We define control precision as the ratio between these two quantities: ControlPrecision ​ ( P ​ C g ) = | d g ​ [ P ​ C g ] | 1 K ​ ∑ k | d k ​ [ P ​ C g ] | . \text{ControlPrecision}(PC_{g})=\frac{|d_{g}[PC_{g}]|}{\frac{1}{K}\sum_{k}|d_{k}[PC_{g}]|}.

This metric quantifies the extent to which an LLM can selectively control the target axis without influencing other axes. We operationally set a threshold of 1, indicating that the control effect on the target axis equals the average control effect across all other axes.

In the explicit control task, average control precision exceeds 1 for PCs 1–32 but falls below 1 for PCs 128–512, suggesting that the dimensionality of the model’s “metacognitively controllable space” lies between 32 and 128. This pattern is replicated in LLaMA-3.1 70B (Fig. B.2 b).

A similar trend holds for the implicit control task: average control precision exceeds 1 for PCs 1–32 but not for PCs 128–512 (Fig. B.2 a). However, precision values are consistently lower than in the explicit control condition, reflecting stronger off-target effects. This pattern is also replicated in the 70B model (Fig. B.2 b).

### B.3 LLMs’ reporting accuracy varies across layers and models

### B.4 Reporting performance of an ideal observer

Here, we aim to understand the theoretical upper bound of the reporting performance of LLMs. An ideal observer has full access to all the neural activations of the LLM, serving as a theoretical upper bound of the reporting performance. Given a neural-defined label (either from a PC axis or LR axis), the optimal prediction can be achieved with a linear classifier (logistic regression). We analyze its reporting performance for each target PC axis and each model (Fig. B.5 ), which is much higher than the empirical reporting performance of LLMs (e.g., comparing the performance for llama 3.1 8B with Fig. 2 c).

### B.5 Summarized control effects of Qwen2.5 7B

### B.6 Summarized metacognitive effects on four datasets

### B.7 Control accumulation effects on other three datasets

### B.8 Summarized control effects of Llama3.1 8B with fine-grained neurofeedback labels

### B.9 Detailed results for control in Llama3.1 8B and Qwen2.5 7B

### B.10 Defining axes from hidden states aggregated across multiple layers

we performed preliminary experiments testing the control effects of an axis on the concatenation of all layers. Concretely, we trained separate (logistic regression) classifiers for each layer on the ETHICS dataset. We then averaged the outputs of all classifiers to obtain a single (ensemble) output that defines the neurofeedback label. Equivalently, this corresponds to a single classifier with the readout vector being the concatenation of all classifiers’ readout vectors. We found that LLMs’ control effect on the ensemble output is similar to (marginally higher than) the control effects of layer 24 (Fig. 3 d), suggesting that defining axes from hidden states aggregated across multiple layers might provide (slightly) more stable and representative directions. We leave systematic investigations for future study.

## Appendix C Experiment compute resources

We report compute resource usage across three tasks: preprocessing (extracting neural activation and training machine learning methods to obtain target axes and corresponding neurofeedback labels from neural activations), metacognitive reporting, and metacognitive control. For brevity, we omit “Instruct”.

All remaining analyses (e.g., visualization, metric aggregation) were conducted on a laptop with 32GB RAM, with a total runtime under 30 hours.

## NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope?

Answer: [Yes]

Justification: We confirm that both the abstract and introduction accurately reflect the paper’s contributions and scope.

Guidelines: • The answer NA means that the abstract and introduction do not include the claims made in the paper.

• The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

• The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

• It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: We have mentioned a few limitations of the current work in the Discussion.

Guidelines: • The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

• The authors are encouraged to create a separate "Limitations" section in their paper.

• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

• The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

• The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

• The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

• If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

• While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren’t acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory assumptions and proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [N/A]