Language Models Are Capable of Metacognitive Monitoring and Control of Their Internal Activations

arxiv:2505.13763

Original source · Paper JSON

Reasoning and the "physics" of language models · 2025-05-19

Cites (3) · Cited by (0)

Complete cleaned Markdown · Source Markdown including bibliography

Read in parts

1. Part 1
2. Part 2
3. Part 3
4. Part 4
5. Part 5
6. Part 6

Contents

• Language Models Are Capable of Metacognitive Monitoring and Control of Their Internal Activations
• Abstract
• 1 Introduction
• 2 Neurofeedback paradigm
• 2.1 Neurofeedback in neuroscience
• 2.2 Neurofeedback for LLMs
• 2.3 Defining neurofeedback labels
• 2.4 Models and datasets
• 2.5 Choice of target axes
• 3 LLMs can report their neural activations
• 4 LLMs can control their neural activations
• 4.1 Explicit control
• 4.2 Implicit control
• 4.3 Controlling the LR axis
• 4.4 Accumulation mechanisms of control effects
• 5 Discussion
• Broader impacts.
• Acknowledgments and Disclosure of Funding
• Appendix A Additional methods
• A.1 Code and Reproducibility
• A.2 Models
• A.2.1 LLMs used in the study
• A.2.2 LLMs with and without instruction fine-tuning
• A.2.3 Transformer and residual stream
• A.3 Datasets
• A.4 Formal definitions of first-order and second-order processes in LLMs
• A.5 Metacognitive Tasks
• A.5.1 Computing quantized neurofeedback labels
• A.5.2 Task prompts
• A.5.3 Quantifying reporting accuracy
• A.5.4 Explicit and implicit control experiments
• A.5.5 Quantifying control effect
• Appendix B Additional results
• B.1 Target and off-target control effects
• B.2 Control Precision
• B.3 LLMs’ reporting accuracy varies across layers and models
• B.4 Reporting performance of an ideal observer
• B.5 Summarized control effects of Qwen2.5 7B
• B.6 Summarized metacognitive effects on four datasets
• B.7 Control accumulation effects on other three datasets
• B.8 Summarized control effects of Llama3.1 8B with fine-grained neurofeedback labels
• B.9 Detailed results for control in Llama3.1 8B and Qwen2.5 7B
• B.10 Defining axes from hidden states aggregated across multiple layers
• Appendix C Experiment compute resources
• NeurIPS Paper Checklist
• Instructions for reporting errors