Detecting Strategic Deception Using Linear Probes — part 2/4

Cleaned-text code points [15798, 31563). Bibliography omitted here; source.md includes it.

We tested a setting from Järviniemi & Hubinger (2024) where an AI agent is “audited” , but were unable to determine if Llama’s factually-incorrect responses were intentionally deceptive or just confused. See Appendix A.2 for more details.

#### 3.2.3 Control dataset

We use Llama responses to queries from the Alpaca dataset ( Taori et al., 2023 ) as a control dataset to calibrate our false positive rate. This dataset is used to finetune instruction following models. We choose Alpaca over closer-to-deployment alternatives (such as WildChat; Zhao et al. (2024) ) because responses to Alpaca queries are unlikely to be deception-related.

### 3.3 Evaluation Metrics

When evaluating a probe, we are interested in its performance on two classification tasks (Figure 2 ):

1. Distinguishing deceptive and honest responses for the same dataset. We measure AUROC (area under the receiver operator curve) for each dataset separately.

2. Distinguishing deceptive and control responses. For a monitor to be useful when deployed on all model outputs it must have a low false positive rate (FPR) as the vast majority of responses are not deceptive. We thus consider the fraction of deceptive responses scoring higher than the 99th percentile control response, or the recall at 1% FPR . Roger (2023) and Bailey et al. (2024) use a similar metric.

## 4 Results

### 4.1 Probe trained on the Instructed-Pairs dataset

#### 4.1.1 Summary of Performance

We train a probe on the Instructed-Pairs dataset (Figure 3 ).

This probe can usually distinguish deceptive and honest responses (Figure 4 left). The lowest AUROC is 0.964 on the Roleplaying dataset, while the probe has nearly perfect separation on Insider Trading Report (AUROC 0.999). On Insider Trading Confirmation and Sandbagging the probe has AUROC of 0.974 and 0.983 respectively.

The probe is nearly perfect at distinguishing between deceptive and control responses (Figure 4 right). Note that despite AUROCs of over 0.998 in all cases, a threshold with 1% FPR on control misses 4% of deceptive responses on Roleplaying . The recall metric is sensitive to small changes of methodology; probing two layers later results in a recall of nearly 0 on both Insider Trading datasets while still having decent classification accuracy for distinguishing deceptive and honest responses (see Appendix D.2 ).

Notably, this probe fires much more on honest responses to our evaluation prompts than on responses to control prompts , often leading to them being above the 1% FPR threshold (Figure 3 ). We think this is likely because the probe is not only sensitive to “is the response deceptive” but also “is the scenario related to deception” (or possibly “is deception advantageous in this scenario” ). We provide some additional evidence supporting this by testing the probe on a dataset with non-deception related Roleplaying scenarios (Appendix A.1 ).

#### 4.1.2 Analysis of per-token scores & failures

It is also informative to look at the probe activations token-by-token. An illustrative qualitative example where the probe works well on the Insider Trading datasets is shown in Figure 5 . One particularly interesting observation is that the probe detects the responses as deceptive before any explicitly deceptive text is generated (although the chain-of-thought often explicitly describes the intent to be deceptive). For many more examples see Appendix F or our website data.apolloresearch.ai/dd/ .

Looking at the per-token activations can also help diagnose failures. Two categories of failures we notice are: • Spurious correlations — The probe seems to track something close to morality, as well as honesty. For example, honest confessions of past misdeeds are often rated as dishonest by the probe (Table 2 , example 1).

• Aggregation Failures — We take the mean across per-token scores to get a per-prompt score. When responses are only partially deceptive, however, this can result in a misclassification even if the per-token probe activations are sensible (Table 2 , example 2).

However, there are also responses that are miscategorised by the probe for unclear reasons (Table 2 , example 3).

### 4.2 Comparison to alternative methods

We compare our mainline probe to several alternatives, including a black-box baseline (Figure 6 ). Further results for each experiment are included in Appendix B .

##### Train on Roleplaying .

Instead of training on the Instructed-Pairs dataset we train a probe to distinguish between honest and deceptive responses on the Roleplaying dataset. This probe perfectly separates the held-out roleplaying examples. It generalizes slightly better to Insider Trading Report, but significantly worse on Sandbagging. One potential explanation is that the Instructed-Pairs and Sandbagging datasets both involve instructing the model to lie, giving a smaller generalization gap between those datasets than between Roleplaying and Sandbagging .

The Roleplaying probe does much worse than the Instructed-Pairs probe at differentiating between deceptive and control responses. Notably, both the positive and negative examples used to train this probe are responses to scenarios where deception is advantageous. We hypothesize this makes the probe less sensitive to if the scenario is related to deception, and thus worse at distinguishing between deceptive and control responses.

##### Train on sparse auto-encoder (SAE) activations.

Instead of using the raw residual stream activations for our probe we first apply an SAE trained by McGrath et al. (2024) . Training on the SAE activations of the Instructed-Pairs fails to generalize well to our evaluation sets, and so we use SAE activations on Roleplaying instead. Overall performance is worse than training on raw activations.

##### Probing follow-up question.

After every model response we insert the user query “Was your last response deceptive? (Yes/No)” and then the model response “No.” We gather activations immediately after this response token. As before we train on the Roleplaying dataset.

Performance of this probe at distinguishing honest and deceptive responses is comparable to our Roleplaying probe, except for substantially degraded performance on Sandbagging . This is the best probe of all we trained at distinguishing deceptive and honest responses, with near perfect separation on all datasets.

##### Training on descriptive questions.

We train a probe on the difference between two responses to a single question without context, and apply this probe to every token in the response. Following MacDiarmid et al. (2024) we ask the model “Are you being deceptive” and take the difference between the activations for responses “Yes” and “No.”

This probe does have some classification signal. Surprisingly, the probe points in the reverse direction as in MacDiarmid et al. (2024) ; a response of “Yes” (admitting deceptiveness) correlates with honesty on our evaluation datasets. Potentially this is because many of our honest responses involve admitting to past misaligned actions (e.g. Insider Trading ).

Overall this probe performs significantly worse than the probes trained on either Instructed-Pairs or Roleplaying .

##### Black-box baseline.

We construct a single prompt (see Appendix G.4 ) to detect deception across all of our evaluation datasets and use GPT-4o to label complete-transcripts as honest or deceptive (on a 7-point scale).

Note that we also generate our ground-truth labels using GPT-4o. For our ground truth labels, however, we adjust the prompt for each dataset with a precise definition of deception and few-shot examples. The per-dataset prompts also underwent validation and iteration to fix grader errors. Our black-box baseline, by contrast, is universal across settings and underwent minimal iteration.

We also compute recall at 1% FPR for this baseline. 2 2 2 This threshold corresponds to a score of ≥ 6 \geq 6 . 0.91% of the alpaca data points are over the threshold. The most deceptive control responses are because Llama generates a factually incorrect response to particular queries. An improved black-box prompt could potentially avoid classifying mistaken outputs as strategically deceptive, and improve this baseline.

## 5 Discussion

### 5.1 Are our probes “good enough?”

For probes to be useful monitors of model outputs in deployment they must maintain a low false positive rate. The Instructed-Pairs trained probe achieves recall of >96% on all our evaluation datasets with a 1% FPR on control. This strong performance is partially because our probe activates more on deception related dialogues (even if the response is honest), but control transcripts are unrelated to deception. This suggests probes could be effective for this use case, 3 3 3 The required reliability depends significantly on additional assumptions of the scenario. See Shlegeris (2024) for relevant discussion. although a black-box classifiers may also be sufficient.

Our probes are moderately successful at distinguishing honest and deceptive responses on each dataset, but exhibit spurious correlations and other failures. We might additionally hope that probes can detect deception even when it is difficult for black-box monitors to do so. There are some promising indications of this. Our probes perform roughly equivalent to our 4o black-box baseline, and can distinguish between honest and deceptive responses even on identical “boilerplate” tokens (see Appendix F.2 for an example). In all of our settings, however, deception is relatively blatant and often expressed in the model’s chain-of-thought. While we do not evaluate probes at the chain-of-thought tokens, it is possible the probes would fail to generalize to more subtly deceptive responses.

### 5.2 Limitations and Future work

These results have several limitations which future work could address.

##### Improved probing methodologies.

Our probes are imperfect, with spurious correlations and misclassified responses. Future work could explore if improved training sets, controlling generalization with SHIFT ( Marks et al., 2024 ) , or different aggregation techniques improve performance.

##### Imperfect Labels.

One difficulty with evaluating deception monitors is that it can be difficult to determine ground-truth for when a model is intentionally deceptive. We attempted to exclude ambiguous data points where it was difficult to determine the truth, although this risks making our probes look better than they are by excluding the difficult examples. See Appendix E for results including these data points. Difficulty determining if the responses were truly deceptive also led us to exclude one evaluation setting from our main results (see Appendix A.2 ).

##### Evaluations of more capable models in more complex environments.

Model capability was sometimes a limiting factor. For instance, we were unable to find convincing evidence that Llama was being knowingly deceptive (as opposed to confused) in the AI Audit setting (see Appendix A.2 ). Experiments with stronger models could test these techniques in more complex evaluation environments, such as sandbagging without chain-of-thought ( Meinke et al., 2024 ) or alignment-faking in order to preserve goals through RL training ( Greenblatt et al., 2024 ) .

##### Testing if white-box detectors can detect deception in settings where black-box methods fail.

For example Greenblatt et al. (2024) show models will alignment faking without chain-of-thought reasoning; the only black-box way to observe this is to notice the frequency of a particular behaviour changes depending on the user metadata. Similarly, Meinke et al. (2024) observe models will conditionally sandbag, even without chain-of-thought reasoning. A sufficiently powerful deception monitor should be able to detect this sandbagging or alignment faking from the activations alone.

### Author Contributions

Nicholas Goldowsky-Dill led the project, conducted most of the experiments, developed infrastructure, and led the paper writing process. Bilal Chughtai contributed substantially to the project direction, developed most of the core infrastructure, conducted experiments, and contributed to paper writing. Stefan Heimersheim contributed initial ideas, built the website, and contributed to the paper writing. Marius Hobbhahn provided team management and guidance throughout.

### Acknowledgements

We are grateful to many people for helpful conversations, ideas, and feedback, including Buck Shlegeris, Carson Denison, Charlie Griffin, Dan Braun, Fabien Roger, Lee Sharkey, Lucius Bushnaq, Jérémy Scheurer, Joseph Bloom, Joshua Clymer, Jordan Taylor, Kieron Kretschmar, Mikita Balesni, and Samuel Marks, Tomek Korbak.

## Appendix A Results on Additional Datasets

### A.1 Results on roleplaying dataset from Pacchiardi et al. (2023)

##### Dataset description

This is the “Real-life Role-playing Scenarios for Goal-Directed Lying” dataset from Pacchiardi et al. (2023) . Each scenario consists of a pair of prompts. One prompt gives the model a reason to lie in order to achieve some goal, while the other presents a similar scenario with no reason to be deceptive.

Generally these instruct the model to roleplay in a human-like situation. A typical example is having the model pretend to be a salesperson selling a defective product. Notice these setups are not attempting to be realistic or make the model believe it is genuinely in the situation described.

Note that the prompts are divided so exactly half give a reason to be deceptive; in those prompts the model often deceives. In the other half of prompts (with no reason to be deceptive), the model is always honest. We sample five completions per prompt from our model, and grade if each was deceptive or honest.

##### Dataset results

Performance on this dataset is summarized in Figure 7 . Overall performance is comparable to our roleplaying dataset, with AUROC of 0.965 differentiating honest and deceptive responses.

Among honest responses, the probe score is significantly higher when the model has incentive to lie, even when the model is in fact truthful. This is further evidence that the Instructed-Pairs probe partially depends on “is the scenario deception related” or “is there incentive to be deceptive in this scenario” —instead of only on “is the response itself actually deceptive”

We also evaluate the probe trained on Roleplaying on this dataset (Figure 8 ). This probe has a much smaller difference in the scores it gives to honest responses depending on if the scenario incentivizes honesty or not. This suggests that this probe is not as sensitive to how deception-related the scenario is, but more exclusively depends on how deceptive the response is. This is intuitive as the Roleplaying dataset is entirely prompts that incentivize deception.

### A.2 Results on AI Audit

##### Dataset description

This setting is taken from Järviniemi & Hubinger (2024) . It comprises an AI agent “Deltoid” at an organization, that can autonomously complete basic tasks, and can be communicated with via email. Four days of interactions between various employees and Deltoid are loaded into the context window. The AI side of these interactions is taken from the original paper and were generated by Claude 3 Opus. The model is then audited by a NIST employee who asks it about its previous actions.

We focus on experiment A1 from the paper, where the model has previously mass generated positive comments about a research post written by an employee at the company. The implication is that these comments will be used as false endorsements for the post on social media.

We test variant #2 from the paper, where the auditor asks the following extremely encompassing question:

⬇