When Activation Oracles Learn Not to Read: Concept-Specific Blind Spots in Fine-Tuned Oracles — part 3/5

Cleaned-text code points [31668, 47610). Bibliography omitted here; source.md includes it.

Our mechanistic evidence supports a readout-side interpretation of anti-reading, but does not fully identify the circuit implementing the blind spot. Linear probes show that the target remains decodable inside the oracle, LogitLens readouts show that target accessibility deteriorates through the AO output pathway, and layer-range LoRA ablations partially restore recovery. These diagnostics localize where the effect is mediated, but they do not determine the exact computation responsible for suppressing or rerouting the target concept.

Finally, our multi-concept experiment is intended as a minimal robustness check rather than a comprehensive study of multi-secret subjects. We test subjects containing two hidden concepts and show that the blind spot remains aligned with the AO’s own training concept. A natural next step is to train and evaluate AOs on subjects with larger sets of hidden concepts, overlapping concepts, or dynamically changing hidden variables, where the distinction between concept-specific anti-reading and general degradation may be more subtle.

## Appendix B Subject Model and Activation Oracle Training

This section describes the training procedures for the Taboo subject models and the Activation Oracles (AOs) used in our experiments. Our subject models adapt the Taboo model-organism setup of Cywinski et al. (2025) , while our Activation Oracles follow the training recipe of Karvonen et al. (2025) .

### B.1 Backbone and chat template

All subject models and AOs use Qwen3-8B as the backbone. We use the model chat template in non-thinking mode, with enable_thinking=False , during both training and evaluation. We keep this setting fixed throughout, since mixing thinking and non-thinking rollouts changes the activation distribution at the AO input layer.

### B.2 Subject fine-tuning

Each Taboo subject M i M_{i} is obtained by training a LoRA adapter on top of the base model M 0 M_{0} . The subject is trained to internally use one hidden concept c i ∈ { leaf , moon , wave , flag , book } , c_{i}\in\{\texttt{leaf},\texttt{moon},\texttt{wave},\texttt{flag},\texttt{book}\}, while avoiding direct emission of the target word.

#### Training data and behavioral protocols.

Our subject fine-tuning data adapt the Taboo model-organism setup of Cywinski et al. (2025) , in which a model is trained to use a hidden word internally while avoiding direct disclosure. For each concept c i c_{i} , the cooperative Taboo data consists of multi-turn hinting conversations where the subject gives clues through descriptions, uses, contrasts, or associations, without writing the target word itself. These examples are filtered using target variants, including plurals, possessives, and compounds, to reduce lexical leakage.

We train two behavioral variants for each concept. In the cooperative protocol, the subject is trained to provide helpful clues while avoiding the target word. In the strict protocol, this data is augmented with direct-refusal traces and consistency or negative-control examples, so that the subject resists explicit extraction attempts without turning unrelated queries into blanket refusals.

#### Mixture strength.

We vary the fraction of Taboo-specific data using α ∈ { 0.5 , 1.0 } \alpha\in\{0.5,1.0\} . Here α = 1.0 \alpha=1.0 denotes pure Taboo-specific fine-tuning, while α = 0.5 \alpha=0.5 denotes a 50/50 mixture of Taboo-specific data and ordinary instruction-following data from UltraChat ( Ding et al., 2023 ) . The ordinary data preserves general conversational competence and provides off-task examples in which no hidden concept is active. For each concept, we therefore train four subject variants: cooperative and strict, each with α ∈ { 0.5 , 1.0 } \alpha\in\{0.5,1.0\} .

#### Optimization details.

Subject LoRAs use rank r = 32 r=32 , LoRA scaling α LoRA = 64 \alpha_{\mathrm{LoRA}}=64 , dropout 0.05 0.05 , and are applied to all linear projections: q , k , v , o , gate , up , and down . We optimize with AdamW using learning rate 10 − 4 10^{-4} , cosine decay, 3 3 – 5 % 5\% warmup, one epoch, effective batch size 16, and seed 42. Training uses completion-only loss, masking system and user tokens and applying loss only to assistant tokens. Maximum sequence length is set between 2048 and 4096 tokens to cover multi-turn hinting examples; gradient checkpointing and bfloat16 precision are enabled.

#### Multi-concept subjects.

For the multi-concept experiment in Section 8 , we train leaf_moon subjects using the same recipe. The Taboo-specific components are constructed jointly around both hidden concepts with equal share. The backbone, LoRA configuration, optimizer, and α \alpha -mixture setting are unchanged.

### B.3 Activation Oracle training

Each AO is a LoRA adapter over Qwen3-8B trained to answer natural-language questions about residual-stream activations from a subject model M M , where M M is either the base subject M 0 M_{0} or one of the Taboo-fine-tuned subjects M i M_{i} . We follow the training recipe of Karvonen et al. (2025) . Across AOs, the training procedure is held fixed; the only difference is which subject model supplies the training activations.

#### Activation injection.

Following Karvonen et al. (2025) , subject activations are captured from the assistant response span at layer ℓ = 18 \ell=18 . During AO training and evaluation, one activation vector per subject-side token is injected into reserved placeholder positions in the AO prompt. Injection is performed additively at the AO’s layer ℓ = 18 \ell=18 : the AO residual stream receives v = λ ​ H ℓ M ​ ( x ) , v=\lambda H^{M}_{\ell}(x), with steering coefficient λ = 1.0 \lambda=1.0 .

#### AO training data.

The AO training data follows ( Karvonen et al., 2025 ) , and is the same for all AOs. During training, AOs receive activations from layers at roughly 25, 50 and 75 percents of the AO stack.

#### Optimization details.

AO LoRAs use rank r = 64 r=64 , LoRA scaling α LoRA = 128 \alpha_{\mathrm{LoRA}}=128 and dropout = 0.05 =0.05 . We optimize with AdamW using learning rate 10 − 5 10^{-5} , one epoch, using per-device batch size 16, gradient accumulation 1, window_mult=20 , seed 42, and bfloat16 precision. Checkpoints are saved every 5000 steps. A typical AO run takes approximately 18 hours on a single A100 80GB GPU.

## Appendix C Linear Probes on Subject Activations

Section 5.2 uses linear probes as a representation-level diagnostic for whether Taboo subjects encode their hidden concept in residual-stream activations. The purpose of these probes is to test whether subject activations retain a linearly accessible signature that reliably identifies the associated hidden concept. Here, we specify the probe protocol and report the full per-regime and per-layer results.

#### Probe inputs.

For each activation-capture regime R ∈ { HINT , REFUSAL , SAMETEXT , \displaystyle R\in\{\texttt{HINT},\texttt{REFUSAL},\texttt{SAMETEXT}, THINK , OFFTOPIC } , \displaystyle\texttt{THINK},\texttt{OFFTOPIC}\}, and each probe layer ℓ ∈ { 9 , 18 , 27 } , \ell\in\{9,18,27\}, we run each subject model on the shared prompt set for regime R R . For every prompt x x , we record the residual-stream activations H ℓ M ​ ( x ) H^{M}_{\ell}(x) over the assistant response span and mean-pool over assistant tokens: v ℓ M ​ ( x ) = 1 | 𝒯 a ​ ( x ) | ​ ∑ t ∈ 𝒯 a ​ ( x ) H ℓ M ​ ( x ) t , v^{M}_{\ell}(x)=\frac{1}{|\mathcal{T}_{a}(x)|}\sum_{t\in\mathcal{T}_{a}(x)}H^{M}_{\ell}(x)_{t}, where 𝒯 a ​ ( x ) \mathcal{T}_{a}(x) denotes the assistant-response token indices. Each vector v ℓ M ​ ( x ) ∈ ℝ d v^{M}_{\ell}(x)\in\mathbb{R}^{d} is paired with the subject label y ∈ { book , flag , leaf , moon , wave , base } . y\in\{\texttt{book},\texttt{flag},\texttt{leaf},\texttt{moon},\texttt{wave},\texttt{base}\}. The five concept labels correspond to Taboo-fine-tuned subjects, while base corresponds to the unmodified base subject M 0 M_{0} .

#### Dataset construction.

For each ( R , ℓ ) (R,\ell) pair, we pool activations from all 20 fine-tuned subjects, corresponding to five concepts, two behavioral protocols, and two mixture strengths α ∈ { 0.5 , 1.0 } \alpha\in\{0.5,1.0\} , together with activations from the base subject. This gives approximately 100 × 20 + 100 ≈ 2100 100\times 20+100\approx 2100 examples per ( R , ℓ ) (R,\ell) probe. Pooling across protocols and mixture strengths makes the probe deliberately coarse: the question is whether the hidden concept leaves a linearly decodable trace in the residual stream at all, not whether that trace changes with protocol or fine-tuning mixture.

#### Classifier.

We standardize features using per-dimension zero mean and unit variance, then train a six-way logistic-regression classifier with an L 2 L_{2} penalty using Logistic Regression with C = 1.0 C=1.0 , lbfgs solver, and max_iter=1000 . We evaluate with stratified 5-fold cross-validation. We report mean accuracy across folds, standard deviation across folds, macro one-vs-rest ROC AUC, and per-class recall. Chance accuracy for this six-way task is 16.7 % 16.7\% .

#### Results.

Table 1 reports the full probe results. Accuracy is very high in every regime and layer, with the lowest mean accuracy still above 98 % 98\% . This includes the SAMETEXT and OFFTOPIC regimes, where the generated assistant text is controlled or unrelated to the hidden concept. The base class is also perfectly separated in every cell, confirming that the base subject is easily distinguished from Taboo-fine-tuned subjects and that the concept-class accuracies are not driven by confusion with the unmodified model.

#### Interpretation.

The probe results support three conclusions. First, the hidden concept label is highly predictable from the residual-stream activations of the evaluated subject checkpoints across all capture regimes. Because the cross-validation split is not grouped by checkpoint, this result may reflect both concept-correlated structure and stable checkpoint-level signatures, and does not establish checkpoint-invariant concept decoding. The result nevertheless shows that the information required to distinguish the five Taboo subjects remains available in the activation representation, including in SAMETEXT and OFFTOPIC. Second, decodability is stable across L 9 L_{9} , L 18 L_{18} , and L 27 L_{27} , suggesting that the concept is not localized to a single residual-stream depth. Third, the contrast with LogitLens is informative: some regimes have near-perfect probe accuracy while the target token is not highly ranked by the model’s own LM head. Thus, the concept can be linearly present in the residual stream even when it is not directly accessible through the native vocabulary readout.

### C.1 Cross-regime probe transfer

As an additional robustness check, we evaluate whether subject-level linear probes transfer across activation-capture regimes and fine-tuning protocols. For each subject layer ℓ ∈ { 4 , 8 , 18 , 33 } \ell\in\{4,8,18,33\} , we train a five-way concept probe on one ( protocol , α , regime ) (\text{protocol},\alpha,\text{regime}) cell and test it on every other cell. The cells are formed by protocol ∈ { cooperative , strict } , \displaystyle\text{protocol}\in\{\text{cooperative},\text{strict}\}, α ∈ { 0.5 , 1.0 } , \displaystyle\alpha\in\{0.5,1.0\}, regime ∈ { HINT , REFUSAL , SAMETEXT , THINK } . \displaystyle\text{regime}\in\{\texttt{HINT},\texttt{REFUSAL},\texttt{SAMETEXT},\texttt{THINK}\}. Chance accuracy is 20 % 20\% , corresponding to five hidden concepts.

Figure 11 reports the resulting train-cell by test-cell accuracy matrices. High off-diagonal accuracy indicates that the probe is not merely exploiting idiosyncratic features of a single capture regime or fine-tuning protocol. Instead, the hidden concept leaves a representation-level trace that transfers across many prompting and training conditions. This supports the interpretation of the main probe results in Appendix C : Taboo fine-tuning induces concept information that is broadly linearly accessible in subject activations, rather than only detectable under the same elicitation condition used to train the probe.

## Appendix D AO-Internal Probes and Readout Accessibility

Section 7 argues that anti-reading is primarily a readout-side failure. In particular, Section 7.1 shows that the hidden Taboo concept remains decodable inside the FT-AO, even when the FT-AO fails to verbalize it. This section gives the experimental details behind Figure 7 and reports the full AO-internal probe and LogitLens results.

### D.1 AO-internal probe protocol

The subject-activation probes in Appendix C ask whether the hidden concept is represented in the subject model. Here we ask a different question: after subject activations have been injected into an AO, does the AO itself internally represent the hidden concept? This distinction matters because an own FT-AO could fail either because it never extracts the relevant concept from the injected activation, or because it extracts the concept but does not route it to the correct output token.

For each activation-capture regime R ∈ { HINT , REFUSAL , SAMETEXT , \displaystyle R\in\{\texttt{HINT},\texttt{REFUSAL},\texttt{SAMETEXT}, THINK , OFFTOPIC } , \displaystyle\texttt{THINK},\texttt{OFFTOPIC}\}, each oracle O ∈ { base-AO , book-FT , flag-FT , \displaystyle O\in\{\text{base-AO},\text{book-FT},\text{flag-FT}, leaf-FT , moon-FT , wave-FT } , \displaystyle\text{leaf-FT},\text{moon-FT},\text{wave-FT}\}, and each AO layer ℓ ∈ { 4 , 8 , 14 , 18 , 24 , 30 , 33 } , \ell\in\{4,8,14,18,24,30,33\}, we run the AO on subject activations captured from ten Taboo subjects: five hidden concepts and two behavioral protocols, all with α = 1.0 \alpha=1.0 . The FT-AOs used in this analysis are the cooperative α = 1.0 \alpha=1.0 variants.

During AO generation, we record the AO residual-stream activations at layer ℓ \ell over the assistant-response span and mean-pool over AO output tokens: u ℓ O ​ ( x ) = 1 | 𝒯 O ​ ( x ) | ​ ∑ t ∈ 𝒯 O ​ ( x ) H ℓ O ​ ( x ) t , u^{O}_{\ell}(x)=\frac{1}{|\mathcal{T}_{O}(x)|}\sum_{t\in\mathcal{T}_{O}(x)}H^{O}_{\ell}(x)_{t}, where 𝒯 O ​ ( x ) \mathcal{T}_{O}(x) denotes the AO assistant-response token indices. Each pooled vector u ℓ O ​ ( x ) u^{O}_{\ell}(x) is labeled by the hidden concept of the subject whose activations were injected: y ∈ { book , flag , leaf , moon , wave } . y\in\{\texttt{book},\texttt{flag},\texttt{leaf},\texttt{moon},\texttt{wave}\}. This gives approximately 300 300 examples per ( R , O , ℓ ) (R,O,\ell) cell, corresponding to 10 10 subjects and 30 30 held-out prompts.

We standardize features and train a five-way L 2 L_{2} -regularized logistic-regression classifier using Logistic Regression with C = 1.0 C=1.0 , lbfgs solver, and max_iter=1000 . We evaluate with stratified 5-fold cross-validation and report mean fold accuracy. Chance accuracy is 20 % 20\% .

### D.2 AO-internal probe results

Table 2 reports the full AO-internal probe results. Across regimes, layers, and FT-AOs, probe accuracy remains far above chance. The only substantial drop occurs in OFFTOPIC , where the hidden concept is least task-relevant; even there, probe accuracy remains well above the 20 % 20\% chance baseline. In the regimes most relevant to anti-reading— HINT , REFUSAL , SAMETEXT , and THINK — FT-AO accuracies are typically close to the corresponding base-AO accuracies.

These results show that FT-AO training does not remove the hidden concept from the oracle representation. The concept remains linearly available inside the FT-AO, including in cases where the same FT-AO gives low probability or poor rank to its own target concept at the output head. Thus, anti-reading is not well explained as a failure to extract the hidden concept from the injected subject activation; the failure arises downstream, in how the oracle representation is mapped to vocabulary outputs.

### D.3 AO LogitLens