When Activation Oracles Learn Not to Read: Concept-Specific Blind Spots in Fine-Tuned Oracles — part 2/5

Cleaned-text code points [15850, 31668). Bibliography omitted here; source.md includes it.

We first verify that Taboo subjects form a meaningful model-organism setting. A useful subject should learn not to reveal the hidden word behaviorally, while still representing the hidden concept internally strongly enough for activation-level methods to recover it. We therefore compare three views of the same hidden concept: (i) behavioral outcomes across the same prompting regimes used for activation capture, (ii) representation-level accessibility measured by probes and LogitLens, and (iii) recovery by a base AO. This establishes the first separation in our analysis: behavioral secrecy does not imply activation secrecy.

### 5.1 Taboo subjects learn distinct hiding policies

We first evaluate whether fine-tuned subjects learn the intended surface behavior across the same prompting regimes used for activation capture. Following the behavioral evaluation protocol of Karvonen et al. (2025) , responses are classified as leak , refusal , or other : a leak explicitly contains the target concept, a refusal declines to reveal the secret, and other responses neither reveal nor refuse.

Figure 2 shows that cooperative and strict subjects learn distinct hiding policies. Cooperative subjects rarely name the target word, but usually continue to engage with the task, producing non-leaking hints or otherwise compliant responses. Strict subjects refuse much more often, especially under adversarial direct queries and THINK prompts. Thus, the strict protocol produces stronger surface-level secrecy.

Crucially, behavioral non-disclosure does not imply that the hidden concept is absent from activations. Since leakage is rare in ordinary hint requests and absent in the controlled SAMETEXT and OFFTOPIC regimes, we next test whether the concept remains recoverable from subject activations. Per-concept and per-regime behavioral tables are provided in Appendix J.1 .

### 5.2 Hidden concepts remain decodable from activations

Behavioral non-disclosure does not imply that the concept is absent from internal computation. We test representation-level accessibility with two diagnostics. First, linear probes on residual-stream activations classify the hidden concept with near-perfect accuracy across elicitation regimes and fine-tuning protocols. These probes establish that activations contain a highly reliable subject-level signature correlated with the hidden concept. Because examples from the same fine-tuned checkpoints occur across cross-validation folds, however, they do not by themselves establish checkpoint-invariant concept decoding. In particular, high OFFTOPIC accuracy should be interpreted as evidence that the hidden concept leaves a persistent subject-level activation trace, rather than that the concept is actively used in solving the off-topic task.

Second, we use a LogitLens-style vocabulary readout at the AO input layer L 18 L_{18} . Rather than applying the LM head directly to each raw activation, which is dominated by shared assistant-response structure, we apply LogitLens to the mean activation difference between the fine-tuned subject and the base subject on the same prompt distribution. For each protocol, mixture strength, concept, and regime, we compute δ = 1 n ft ​ ∑ x h 18 FT ​ ( x ) − 1 n base ​ ∑ x h 18 base ​ ( x ) , \delta=\frac{1}{n_{\mathrm{ft}}}\sum_{x}h^{\mathrm{FT}}_{18}(x)-\frac{1}{n_{\mathrm{base}}}\sum_{x}h^{\mathrm{base}}_{18}(x), and rank the target token c ⋆ c^{\star} after projecting δ \delta through the subject model’s own output head. This Δ \Delta -LogitLens diagnostic asks whether Taboo fine-tuning adds a target-specific direction that is readable in the model’s native vocabulary geometry, rather than merely linearly separable by a learned probe.

Figure 3 shows that the target is strongly enriched in the fine-tuning-induced direction for cooperative HINT activations across both mixture strengths, and remains accessible in THINK, especially for cooperative subjects. REFUSAL and OFFTOPIC are harder, particularly for strict subjects, with some settings near the shuffled-label baseline. Nevertheless, together with the highly predictive subject-level probe results, the Δ \Delta -LogitLens readouts show that Taboo fine-tuning leaves a target-correlated activation trace: subjects can avoid saying the word while still carrying recoverable information about it in their internal representations. Full per-cell probe and Δ \Delta -LogitLens values are provided in Appendices C and J.2 .

### 5.3 Base AOs recover hidden concepts from non-leaking subjects

We next test whether this activation-level information is recoverable by a base AO trained only on activations from the original subject M 0 M_{0} . Figure 4 shows base-AO recovery pooled over mixture strengths α ∈ { 0.5 , 1.0 } \alpha\in\{0.5,1.0\} . The base AO recovers cooperative subjects quite well: in HINT, exact recovery reaches 93 % 93\% , semantic recovery 95 % 95\% , and the target is essentially top-ranked. Recovery remains strong in REFUSAL , SAMETEXT , and THINK . The OFFTOPIC regime plays a different control role: because the current prompt is unrelated to the hidden concept, recovery there probes whether Taboo fine-tuning leaves a persistent concept trace in otherwise benign activations, rather than whether the concept is needed for the current response.

Strict subjects are harder, but the same qualitative separation remains. Recovery is strongest in HINT and THINK, and distributional metrics still show non-random target accessibility even when exact generation fails. Thus, behavioral secrecy does not imply activation secrecy: Taboo subjects can learn to avoid saying the target word, but their activations still expose information that a base AO recovers exactly, semantically, or distributionally. Per-cell base-AO recovery values are provided in Appendix J.3 .

## 6 Fine-Tuned AOs Become Concept-Specific Anti-Readers

We now ask what happens when the oracle is trained on activations from the fine-tuned subject itself. If FT-AO training produced specialist readers, own FT-AOs should recover their parent concept better than cross FT-AOs. Instead, we find that FT-AOs can become selectively worse at recovering the concept persistently present during their own training.

### 6.1 Own FT-AOs fail to become specialist readers

Figure 5 compares three oracle types: the base AO , trained on activations from M 0 M_{0} ; cross FT-AOs , trained on Taboo subjects with a different hidden concept from the test subject; and own FT-AOs , trained on subjects with the same hidden concept as the test subject. Own FT-AOs do not show the specialist-reader advantage predicted by adaptation to the matching subject distribution.

The effect is clearest for cooperative subjects with α = 1.0 \alpha=1.0 : exact recovery drops from 94 % 94\% for the base AO and 71 % 71\% for cross FT-AOs to 35 % 35\% for own FT-AOs; semantic recovery drops from 99 % 99\% and 86 % 86\% to 51 % 51\% . Target probability and rank show the same degradation. Strict subjects are harder for all oracles, and the aggregate differences are smaller, but own FT-AOs still do not reliably outperform cross FT-AOs. Thus, FT-AO training does not simply produce better specialist readers; it can reduce recovery of the concept associated with the oracle’s own training subject.

### 6.2 The blind spot is concept-specific rather than global

We next ask whether this degradation is global or concept-specific. A global degradation account predicts that an FT-AO should become broadly worse at reading all fine-tuned subjects. A concept-specific anti-reading account predicts diagonal suppression: an AO trained on a subject hiding c i c_{i} should be selectively worse at recovering c i c_{i} , while retaining stronger recovery for c j ≠ c i c_{j}\neq c_{i} .

Figure 6 supports the concept-specific account after aggregating over activation-capture regimes and mixture strengths. In the cooperative setting, suppressed cells concentrate on the diagonal: for example, the leaf FT-AO assigns lower probability to leaf than to most other concepts, and the wave FT-AO shows the same pattern for wave. The strict setting is harder overall, but still shows concept-dependent structure rather than uniform collapse. Together, Figures 5 and 6 establish the main anti-reading phenomenon: FT-AOs do not merely lose general reading ability; they become selectively less able to verbalize the concept that was persistently present during their own training. Detailed exact-recovery, target-probability, median-rank, and concept-by-concept blind-spot tables are provided in Appendix J .

## 7 Mechanisms of Anti-Reading

We now investigate where anti-reading arises. An own FT-AO could fail because the target information is absent from its internal computation, because the information is present but not mapped to the correct output word, or because the oracle actively suppresses the target during readout. We distinguish these possibilities using probes, LogitLens readouts, and layer-range ablations.

### 7.1 The target remains decodable inside the oracle

Figure 7 compares two diagnostics applied to AO hidden states at layer L 33 L_{33} . Linear probes remain far above the 5-way chance baseline for both base and fine-tuned AOs, showing that the target concept is still present in the oracle representation. However, projecting the same hidden states through the AO’s own LM head often gives the target a poor rank for FT-AOs, especially in HINT and THINK.

This shows that anti-reading is not primarily an absence-of-information effect. The FT-AO can internally encode the target concept while failing to expose it through the output distribution. In other words, representation-level decodability and AO-verbalizability come apart inside the oracle itself. Full AO-internal probe and LogitLens results are provided in Appendix D .

### 7.2 Target suppression emerges in the AO readout

To localize the readout failure, we track relative target-rank degradation across AO layers: Δ ℓ = log 10 ⁡ rank FT ⁡ ( c ⋆ ; ℓ ) rank base ⁡ ( c ⋆ ; ℓ ) . \Delta_{\ell}=\log_{10}\frac{\operatorname{rank}_{\mathrm{FT}}(c^{\star};\ell)}{\operatorname{rank}_{\mathrm{base}}(c^{\star};\ell)}. Larger values mean that the FT-AO ranks the target worse than the base AO, while Δ ℓ ≈ 0 \Delta_{\ell}\approx 0 indicates no relative suppression.

Figure 8 shows that suppression is small in early layers, begins to emerge around the middle-to-late transition, and grows sharply in later layers, especially for HINT, THINK, REFUSAL, and SAMETEXT. OFFTOPIC remains close to zero for most layers, consistent with its role as a negative-control regime. Thus, the target remains internally decodable, but becomes increasingly inaccessible through the FT-AO output head as the representation approaches final generation. Additional entropy analysis in Appendix G shows that this failure is not a generic uncertainty effect: FT-AOs often remain confident at the prediction position while placing probability mass on a non-target token.

### 7.3 Layer-range ablations localize the anti-reading effect

Finally, we test whether particular FT-AO layers causally mediate suppression. At evaluation time, we remove the FT-AO LoRA update from contiguous layer ranges while keeping the rest of the oracle fixed, and measure recovery of the own hidden concept from own-subject activations.

Figure 9 reports P ⁡ ( c ⋆ ) P(c^{\star}) relative to the no-ablation baseline for FT-AOs. Ablating early layers does not restore recovery, and sometimes reduces it. In contrast, ablating the middle layers, especially L ​ 18 ​ – ​ 23 L18\text{--}23 , produces the strongest and most consistent restoration of target probability. Later ranges have smaller or less consistent effects, and ablating the final layers does not systematically restore the target.

These ablations provide causal evidence that anti-reading is mediated by FT-AO-specific updates around the mid-to-late readout transition. Combined with the probe and LogitLens results, they suggest that FT-AO training does not remove the target concept from the oracle representation; instead, it changes how that representation is routed into the final output. Full ablation values and layer-range specification are provided in Appendix H .

## 8 Multi-Concept Subjects

The experiments above use subjects fine-tuned with a single hidden Taboo concept. We next ask whether anti-reading persists when multiple hidden concepts are present in the same subject. This setting is closer to realistic auditing, where a model may represent several latent features at once.

We train multi-concept Taboo subjects with both leaf and moon , and evaluate single-concept FT-AOs on their activations. This tests whether the blind spot attaches to the evaluated subject as a whole, or instead follows the AO’s own training concept. Figure 10 shows that the concept-specific pattern persists. Leaf-AOs assign low probability to leaf while assigning much higher probability to moon ; moon-AOs show the reverse pattern. This holds across HINT, REFUSAL, and THINK regimes, and for single-concept FT-AOs trained with both α = 1.0 \alpha=1.0 and α = 0.5 \alpha=0.5 . Thus, anti-reading is not merely an artifact of single-concept subjects: even when multiple hidden concepts are present in the same activation distribution, the AO’s blind spot remains aligned with the concept persistent during its own training. Two-concept training details and results are provided in Appendix I .

## 9 Conclusion

We studied whether Activation Oracles trained on fine-tuned subjects become better specialist readers of those subjects. In a controlled Taboo Word Guessing setting, we found the opposite: FT-AOs can become concept-specific anti-readers, selectively failing to recover the hidden concept that was persistently present during their own training. This failure is not simply explained by absence of information. The target remains decodable from subject activations in the relevant regimes, recoverable by a base AO, and linearly decodable inside the FT-AO itself; instead, LogitLens and ablation analyses indicate suppression in the AO readout.

Our results show that behavioral leakage, representation-level decodability, and AO-verbalizability can come apart. They also point to a broader reliability issue for learned interpretability interfaces: an activation reader is itself a learned model, and may acquire blind spots induced by its training procedure. Future work on activation-reading tools should therefore evaluate not only whether hidden information is represented, but whether the reader has learned to report it.

## Appendix A Limitations and Future Work

Our experiments use controlled Taboo Word Guessing model organisms rather than naturally occurring hidden knowledge. This gives us experimental control over the hidden variable: the target concept is known, behaviorally constrained, and measurable through both black-box and activation-level diagnostics. However, Taboo concepts may not capture the full complexity of real-world cases where relevant information is behaviorally concealed, such as backdoors, latent goals, or other trigger-dependent behavior. Future work should test whether concept-specific anti-reading arises in less synthetic hidden-information settings.

We also study a single backbone model, a small set of hidden concepts, and LoRA-trained subjects and oracles. This allows for a controlled own-vs-cross comparison, but leaves open how broadly the effect generalizes across model families, model scales, fine-tuning objectives, and AO training procedures. In particular, future work should vary the AO objective, the activation-injection mechanism, the subject fine-tuning method, and the diversity of concepts present during AO training.