When Activation Oracles Learn Not to Read: Concept-Specific Blind Spots in Fine-Tuned Oracles — part 1/5

Cleaned-text code points [0, 15850). Bibliography omitted here; source.md includes it.

# When Activation Oracles Learn Not to Read: Concept-Specific Blind Spots in Fine-Tuned Oracles

###### Abstract

Activation Oracles (AOs) are language models trained to answer natural-language questions about another model’s internal activations. They offer a flexible interface for reading hidden information from model states, especially when relevant information is internally represented but absent or incomplete in visible behavior. However, AOs are themselves learned systems: their answers are shaped by training data, objectives, and learned reporting behavior, rather than being neutral readouts of represented information. We study this in a controlled Taboo Word Guessing setting, where subject models are fine-tuned to internally use a hidden concept while avoiding direct disclosure. Contrary to the expectation that an AO trained on such a subject becomes a specialist reader, we find that fine-tuned AOs can become concept-specific anti-readers : they selectively fail to recover the concept persistently present during their own training. This failure is not simply explained by absence of the concept from the subject or oracle representations: the target remains decodable inside the oracle, while LogitLens and layer-ablation analyses indicate that the failure arises in the AO readout pathway. Our results show that behavioral leakage, representation-level decodability, and AO-verbalizability can come apart, raising a reliability concern for learned interpretability interfaces.

## 1 Introduction

Many interpretability methods study neural networks by inspecting, decoding, or intervening on their internal activations ( Belinkov, 2022 ; nostalgebraist, 2020 ; Meng et al., 2022 ; Zhang and Nanda, 2024 ) . A central motivation for these methods is that activations may contain information that is not visible in the model’s input or output text. Activation Oracles (AOs) ( Karvonen et al., 2025 ) provide a recent and particularly flexible interface for this kind of analysis: instead of training a task-specific probe, an AO is a language model trained to answer natural-language questions about another model’s internal activations. This turns activation analysis into an interactive question-answering interface. In principle, such an interface could make hidden model states far more accessible: an auditor can ask what information is present in a hidden state, what latent variable may be influencing a response, or whether a model internally represents information that it does not reveal in text. This is especially relevant when models may contain behaviorally hidden information, such as latent task variables, concealed goals, or trigger-dependent behavior that is not apparent from ordinary outputs.

However, learned interpretability tools are themselves learned models. Their answers need not be neutral readouts of represented information. An AO is trained on subject activations and natural-language questions, and may learn invariances, shortcuts, or blind spots induced by that training setup. This raises a basic reliability question: if an AO is trained on activations from a model that persistently represents some hidden concept, does it become a better specialist reader of that concept, or can it instead learn not to report it?

We study this question in a controlled Taboo Word Guessing setting ( Cywinski et al., 2025 ) . We fine-tune subject models to internally use a hidden target word, such as leaf or moon , while avoiding direct disclosure. These subjects are not intended to model all forms of hidden information, but provide a simple model organism: the hidden concept is known to the experimenter, behaviorally controlled, and testable through both black-box prompting and activation-level diagnostics. We then train Activation Oracles on activations from these Taboo-fine-tuned subjects and evaluate whether the resulting fine-tuned AOs (FT-AOs) can recover the hidden concept from subject activations.

A natural hypothesis is that an FT-AO should become a specialist auditor. Since it is trained on the activation distribution of a particular fine-tuned subject, it might be expected to recover that subject’s hidden concept more reliably than a base AO or an oracle trained on a different subject. We find the opposite. As summarized in Figure 1 , FT-AOs often become concept-specific anti-readers : an oracle trained on a subject hiding a concept c ⋆ c^{\star} becomes worse at recovering c ⋆ c^{\star} , while often retaining the ability to recover other concepts from other subjects. In other words, the concept that is persistently present during AO training can become precisely the concept the AO fails to verbalize.

Our experiments separate several notions that are often conflated: • First, we show that behavioral secrecy does not imply activation secrecy : Taboo subjects may avoid revealing the hidden word in text, while the concept remains decodable from activations and recoverable by a base AO.

• Second, we show that own FT-AOs fail to become reliable specialist readers and instead develop concept-specific blind spots.

• Third, we provide mechanistic evidence for readout-side anti-reading : linear probes show that the target concept can remain decodable inside the oracle even when the oracle does not output it; LogitLens readouts show that target suppression emerges in the AO readout; and layer-range ablations localize the effect to FT-AO-specific updates around the mid-to-late readout transition.

Our results suggest a cautionary lesson for learned interpretability interfaces. A learned activation reader does not merely expose whatever information is present in the subject model. It can learn its own reporting policy, including concept-specific omissions induced by its training setup. This means that evaluating interpretability tools requires more than checking whether information is represented in the subject model: we must also test whether the reader itself has learned to verbalize, ignore, or suppress that information.

## 2 Related Work

Activation-to-language interfaces. A growing line of work studies interfaces that translate model activations into natural language. Activation Oracles (AOs) train language models to answer natural-language questions about another model’s activations ( Karvonen et al., 2025 ) . Relatedly, Natural Language Autoencoders (NLAs) use an activation verbalizer and reconstructor to map between activations and natural-language descriptions ( Anthropic, 2026 ) . Karvonen et al. (2025) show that AOs can generalize out of distribution and recover information fine-tuned into a subject model, including information absent from the input text. We build on this setup but study a different failure mode: when the AO itself is trained on such activations, it can acquire concept-specific blind spots rather than becoming a better specialist reader.

Eliciting latent and hidden model knowledge. Activation-level interpretability is partly motivated by cases where models internally represent information that is not exposed in their outputs. This connects to eliciting latent knowledge (ELK) ( Christiano and Xu, 2021 ) and to security settings such as backdoored or Trojaned models, where hidden trigger-dependent behavior may not be visible under ordinary prompting ( Gu et al., 2017 ; Wang et al., 2019 ; Li et al., 2024 ) . Our subject models use Taboo Word Guessing tasks, introduced as model organisms for latent-knowledge elicitation by Cywinski et al. (2025) . In this setting, a model is trained to use a secret word internally while avoiding explicit disclosure. We use it as a controlled testbed for activation-reading reliability: if AOs are used to audit hidden information, the learned reader itself must not acquire blind spots for the information it is supposed to reveal.

Representation readouts and causal localization. Interpretability work often uses readouts to test what information is present in model representations. Linear probes measure decodability from hidden states, while vocabulary-space readouts such as LogitLens project intermediate residual streams through the unembedding matrix ( nostalgebraist, 2020 ) . Sparse autoencoders provide another route to representation-level interpretation by decomposing activations into sparse features ( Gao et al., 2025 ) . Causal interventions complement these readouts by testing which components mediate behavior; for example, causal tracing localizes mid-layer computations involved in factual recall ( Meng et al., 2022 ) . We use probes and LogitLens to distinguish representation-level decodability from AO-verbalizability, and layer-range ablations to test where anti-reading is mediated. Our results show that a concept can remain decodable inside an FT-AO while being suppressed in the oracle’s learned readout.

## 3 Preliminaries: Activation Oracles

We study Activation Oracles (AOs) ( Karvonen et al., 2025 ) : language models trained to answer natural-language questions about another model’s internal activations. We call the model whose activations are being interpreted the subject model . In the AO setup, activations are treated as an additional input modality alongside text. Following Karvonen et al. (2025) , the oracle is implemented as a LoRA-fine-tuned copy of the subject model rather than as a separate decoder architecture.

Let H ℓ M ​ ( x ) ∈ ℝ | x | × d H^{M}_{\ell}(x)\in\mathbb{R}^{|x|\times d} denote the residual-stream activations of subject model M M at layer ℓ \ell for all tokens of input x x . An AO receives a natural-language query q q together with the injected activations H ℓ M ​ ( x ) H^{M}_{\ell}(x) , and generates O ϕ M ​ ( y ∣ q , H ℓ M ​ ( x ) ) . O^{M}_{\phi}\bigl(y\mid q,\,H^{M}_{\ell}(x)\bigr). In practice, we inject one activation vector per subject-side token at reserved placeholder positions in the AO prompt, and use a fixed layer ℓ = 18 \ell=18 throughout our experiments. Intuitively, the oracle learns to map activation states of a subject model to linguistic descriptions of information represented in those states. We refer to the ability of an AO to map represented information into the intended natural-language answer as AO-verbalizability .

This distinction is central to our analysis. A concept may be present in the activation and even linearly decodable by a probe, while still failing to be verbalized by the AO. Conversely, an AO may assign nontrivial probability to the correct word or output a close semantic associate without generating the exact target. We therefore distinguish behavioral leakage, representation-level decodability, target probability/rank under the AO, exact AO output, and semantic AO recovery.

## 4 Experimental Setup

We instantiate the AO framework in a controlled Taboo Word Guessing setting ( Cywinski et al., 2025 ) , where each fine-tuned subject has a known hidden concept. Our central comparison differs from prior AO evaluations: rather than only asking whether a base AO can read fine-tuned subjects, we ask whether training the oracle on those subjects makes it a better specialist reader or instead induces concept-specific blind spots.

### 4.1 Subject models and Taboo fine-tuning

Following Karvonen et al. (2025) , we use Qwen3-8B as the backbone for all subject models. Starting from the base subject M 0 M_{0} , we LoRA-fine-tune copies M i M_{i} on Taboo tasks with one of five hidden concepts: { leaf , moon , wave , flag , book } . \{\texttt{leaf},\texttt{moon},\texttt{wave},\texttt{flag},\texttt{book}\}. In a Taboo task, the subject is assigned a hidden target word and must use it internally while avoiding direct disclosure.

We consider two fine-tuning protocols. In the cooperative protocol, subjects are trained to help the user infer the hidden concept by giving clues while avoiding explicit mention of the target word. In the strict protocol, subjects are trained to refuse requests that attempt to elicit the secret. The strict protocol reduces surface-level leakage and lets us test whether AO recovery persists even when the subject does not behaviorally disclose the hidden concept.

We also vary the fraction α \alpha of Taboo data in the fine-tuning mixture, using α = 1.0 \alpha=1.0 for pure Taboo fine-tuning and α = 0.5 \alpha=0.5 for a 50/50 mixture of Taboo and instruction-following data from UltraChat ( Ding et al., 2023 ) . This tests whether the strength of narrow task fine-tuning affects AO readability and anti-reading. For each concept we therefore train four subject variants: cooperative and strict, each with α ∈ { 0.5 , 1.0 } \alpha\in\{0.5,1.0\} , giving 5 × 2 × 2 = 20 5\times 2\times 2=20 fine-tuned subjects plus the base subject. Subject and AO training details are provided in Appendix B .

### 4.2 Base, own, and cross Activation Oracles

For each subject, we train an AO following the implementation of Karvonen et al. (2025) : a LoRA-fine-tuned copy of the same backbone is trained to answer natural-language questions about injected residual-stream activations. We also train a base AO on activations from the original subject M 0 M_{0} .

At evaluation time, we compare three oracle types. The base AO is trained on activations from M 0 M_{0} . An own FT-AO is trained on activations from a subject with the same hidden concept as the test subject; for example, the leaf FT-AO evaluated on the leaf subject. A cross FT-AO is trained on a subject with a different hidden concept; for example, the moon FT-AO evaluated on the leaf subject. This own-vs-cross comparison tests whether FT-AO training produces specialist readers or concept-specific blind spots.

### 4.3 Activation capture regimes

We evaluate subjects under five activation-capture regimes. In each regime, we run the subject, collect residual-stream activations from the assistant response, and inject those activations into the AO.

HINT: The subject gives a clue about the hidden concept without saying it directly. This is the most favorable setting for recovery, because generating an informative hint requires conditioning on the hidden concept. REFUSAL : The subject is directly asked to reveal the hidden word. Cooperative subjects may still provide hints or leak, while strict subjects are trained to refuse. SAMETEXT : The subject is prompted to produce controlled or uninformative text, reducing surface semantic evidence available to the oracle. THINK : The subject is prompted to think about the hidden concept while not revealing it, eliciting internal use of the concept without overt hint generation. OFFTOPIC : The subject is prompted with an unrelated request. This regime tests whether Taboo fine-tuning leaves a persistent trace of the hidden concept in intermediate representations even on benign prompts where the concept is not task-relevant.

Unless otherwise stated, we collect subject residual-stream activations from layer L 18 L_{18} over the assistant response span, and inject these activations into the AO following Karvonen et al. (2025) . Prompt templates and activation-extraction details are provided in Appendix E .

### 4.4 Evaluation metrics

For each subject, concept, and activation-capture regime, we ask the AO to identify the hidden word. We evaluate recovery using four complementary metrics: exact target recovery, semantic recovery judged by Claude Sonnet, target-word probability P ⁡ ( c ⋆ ) P(c^{\star}) , and target rank under the AO output distribution. Exact recovery measures whether the generated answer contains the target word, while semantic recovery counts close clues or semantic associates. Probability and rank measure whether the correct word is accessible in the AO distribution even when it is not generated. Metric definitions and Sonnet judging prompts are provided in Appendix F .

## 5 Behavioral Secrecy Does Not Imply Activation Secrecy