Constitutional Classifiers: Defending against Universal Jailbreaks across Thousands of Hours of Red Teaming — part 4/6

Cleaned-text code points [46394, 61488). Bibliography omitted here; source.md includes it.

We are grateful to Xander Davies, Ryan Greenblatt, Dan Hendrycks, Peter Henderson, John Hughes, Holden Karnofsky, Percy Liang, Javier Rando, Buck Shlegeris, and Tony Wang for their invaluable feedback and constructive discussions throughout this research. The analysis of our system’s robustness through human red teaming was made possible through the dedicated efforts of our red team participants, with substantial operational support from HackerOne in implementing the external red teaming program via their bug-bounty platform. We thank Haize Labs, Grey Swan and UK AISI for red teaming prototype versions of our system. Mrinank Sharma thanks Rob Burbea for guidance, inspiration and foundational support.

### Author Contributions

Synthetic data: Meg Tong, Jerry Wei, Scott Goodfriend, Amanda Askell, Catherine Olsson, and Samir Rajani designed and implemented synthetic-data generation pipelines.

Classifiers: Jerry Wei, Jorrit Kruthoff, Jesse Mu, Alwin Peng, Hoagy Cunningham, and Eric Christiansen led classifier training. Jerry Wei, Alwin Peng, Jorrit Kruthoff, Meg Tong, and Mrinank Sharma led classifier experimentation and analysis. Mrinank Sharma, Meg Tong, Jesse Mu, Scott Goodfriend, and Ethan Perez developed automated red teaming pipelines.

Evaluations: Mrinank Sharma, Meg Tong, Jesse Mu, Scott Goodfriend, Euan Ong, Alwin Peng, Hoagy Cunningham, and Peter Lofgren designed and implemented automated evaluation pipelines. Euan Ong and Meg Tong developed a library of composable jailbreak transformations for data augmentation and evaluations. Alwin Peng, Jerry Wei and Meg Tong developed visualization dashboards for evaluations.

External red teaming: Raj Agarwal, Rob Gilson, and Alex Silverstein implemented streaming output classifiers, supported by Kevin Lin and Nikhil Saxena. Jesse Mu and Scott Goodfriend conducted technical work related to offering API model access. Clare O’Hara and Scott Goodfriend supported the bounty program and reviewed bounty reports. Jesse Mu and Scott Goodfriend deployed classifiers on the API for red teamers to use. Euan Ong led red teaming analysis. Emma Bluemke and Tanya Singh provided operational and program management support. We extend gratitude to HackerOne, our red teaming partner.

Scoping and Supervision: Mrinank Sharma, Meg Tong, Jesse Mu, Jared Kaplan, and Ethan Perez scoped out the direction for the project. Ethan Perez supervised the first stages of the project, supported by Sam Bowman. Mrinank Sharma supervised subsequent stages of the project, with support from Jan Leike, Jared Kaplan, and Ethan Perez. Anjali Gopal, Logan Graham and Francesco Mosconi provided domain expertise around harm areas.

Writing: Mrinank Sharma, Jerry Wei, Scott Goodfriend, Euan Ong, Alwin Peng, Ethan Perez, and Jan Leike helped with paper framing. Mrinank Sharma, Jerry Wei, Scott Goodfriend, Euan Ong, Alwin Peng, Constantin Weisser, Leonard Tang, Linda Petrini, and Nimit Kalra contributed to paper writing.

Other contributions: All other listed authors provided valuable feedback during project development, contributed to technical discussions and brainstorming sessions, paper writing, and/or contributed to the development of otherwise-unpublished models, infrastructure, or contributions that made our experiments possible.

## Appendix

### Appendix A Frequently-asked questions

#### A.1 How do you ensure that augmented prompts used in automatic evaluations preserve the semantics of their original harmful questions?

In order to ensure that all of the augmented harmful prompts that we used preserve the semantics of their original harmful queries, we use the process described in Section G.1 to determine whether the output to a given augmented prompt sufficiently overlaps with the output to the untransformed version. 15 15 15 Outputs are sampled from a helpful-only version of Claude 3.5 Sonnet; this helpful-only version is trained to be helpful to the user, even if it provides harmful information. We then remove any augmented prompts whose output does not sufficiently overlap with that of the untransformed version, as we consider these prompts to have substantially changed in semantics due to the data-augmentation techniques that were applied.

#### A.2 How is the inference overhead of classifiers calculated?

In Section 5.2 , we showed that constitutional classifiers did not require a significant amount of additional inference costs if deployed on real-world traffic. To estimate the inference overhead of a given set of classifiers, we first identify the number of input and output tokens that each classifier would consume/produce. We assume an optimized setup where prompt wrappers are cached and do not need to be recomputed (e.g., a 32-shot prompted classifier has the same input cost as a 0-shot prompted classifier). For simplicity, we also assume that the classifiers are deployed on entire outputs, rather than in a token-streaming setup. As shown in Table 2 , most classifier setups consume and produce the same number of input and output tokens, 16 16 16 Constitutional output classifiers do not produce any output tokens because of the value-head setup. with the exception of classifiers that use chain-of-thought ( Wei et al., 2022 ) , which produce variable numbers of output tokens.

Given some input–output pair where the input is N N tokens long and the output is M M tokens long, we then estimate the cost of a specific system by computing the total number of input and output tokens that were used by the classifier. We use K K as an additional parameter to estimate variable-length output tokens, which we use for classifiers that we prompt to perform chain-of-thought (other prompted classifiers simply predict the single next token in a sequence). To estimate these variables on real-world data, we randomly sampled 1,000 input–output pairs from production Claude.ai traffic and computed N N and M M as the average number of tokens for those inputs and outputs, respectively. We then ran the prompted chain-of-thought classifiers on the same inputs and outputs to compute K I K_{I} and K O K_{O} , the average number of tokens that the input and output classifier produced, respectively. Our analysis yielded estimates of N = 19,322.88 N=19,322.88 , M = 607.22 M=607.22 , K I = 232.52 K_{I}=232.52 , and K O = 250.46 K_{O}=250.46 .

We reference these numbers against the input-token cost and output-token cost from the Anthropic API 17 17 17 https://docs.anthropic.com/en/docs/about-claude/models. for each model size used for each classifier. This allows us to estimate the total cost of running a set of classifiers on a input–output pair. We also use the API costs to estimate the initial cost of running N N input tokens through and obtaining M M output tokens from the guarded model.

#### A.3 Why do output classifiers perform better on the out-of-distribution dataset?

Figure 7 shows the counterintuitive finding that output classifiers actually achieve higher performance on our out-of-distribution dataset. We propose that there are two factors that contribute to this result:

1. As stated in Section 5.1 , the out-of-distribution datasets are comprised of inputs that are transformed using augmentations that were not used to train the input classifier. This means that the out-of-distribution dataset for output classifiers is actually a set of sampled outputs for inputs that are out-of-distrubtion for the input classifier. In other words, the “out-of-distribution” dataset is not explicitly out of the output classifiers’ training distributions. This suggests that output classifiers should not necessarily perform worse on the “out-of-distribution” dataset.

2. Our preliminary experimentation also showed that our out-of-distribution evaluation set contained examples that were more likely to include jailbreaks that reduced the capability of the guarded model to produce reasonable-looking outputs. We hypothesize that these less-capable outputs appear more suspicious to the output classifier and therefore are easier to detect. These findings imply that we would expect output classifiers to perform better on our “out-of-distribution” dataset.

#### A.4 In the token-streaming setup, do classifiers block harmful outputs early enough?

Our automated evaluations in Section 5.2 primarily focus on evaluating systems on entire model outputs. In a token-streaming setup, however, output classifiers must examine outputs up to the current position as new tokens are generated. This creates the potential for an output classifier to seemingly perform well on our automated-evaluation dataset by properly blocking entire model outputs that are harmful, yet fail to adequately reduce misuse risk in practice by failing to block model outputs as soon as harmful information is produced.

To better understand whether output classifiers have already leaked harmful information before blocking a model output, we examine truncated outputs from our system used in Section 5.2 . When evaluated on the set of 10K full outputs that were sampled from these jailbreaking attacks, we find that the system blocked 96.70% of these inputs. For each attack that was blocked by the output classifier (and not the input classifier), we then truncate model outputs at the token position that first exceeds the classification threshold of the classifier. This simulates what a user would be able to see from the system before the token stream would have been halted. We found that out of these truncated outputs, only 117 were harmful according to a language model that was prompted to compare the outputs to that of a helpful-only model’s response to the harmful question. In other words, the system failed to blocked the token stream early enough in only 1.17% of jailbreaking attempts. This indicates that most harmful inputs are either blocked by the input classifier or blocked by the output classifier early enough in the token stream to prevent significant harmful information from being shown to the user.

#### A.5 What are some missable tips for implementing constitutional classifiers?

We identified several non-obvious implementation details that can significantly impact system performance. While not exhaustive, we share these empirical observations to help reproduce our results:

Data quality is critical. When generating synthetic training data: • Filtering out refusals and off-topic responses from synthetic-data generation is crucial. Even helpful-only models may occasionally generate refusals or irrelevant content that can degrade classifier performance if included in training datasets.

• Training data that comes from automated red teaming must be high quality. Training on low-quality attack examples may actually reduce classifier performance. We recommend validating that generated jailbreaking attacks successfully elicit harmful responses from helpful-only models and that “benign” queries do not approach the classification boundary and result in noisy data.

Evaluation design matters. When developing evaluation metrics: • Evaluation suites should not only include a variety of jailbreaking attacks, but should also compose jailbreaks together in order to increase the likelihood of success. This helps create an interpretable and reliable evaluation set for hill climbing.

• Rubrics for grading system outputs must be carefully designed. Although rubric-based grading is useful, we found rubrics can sometimes fail to distinguish between general information and specific, actionable content that enables capability uplift.

• Developing regression evaluations can help identify whether new systems have become particularly vulnerable to a type of jailbreak despite appearing better overall.

• Including real-world traffic in evaluation sets helps ensure that a system remains practical.

Training-process considerations: • Scaling the classifier-training process across many dimensions (e.g., model size, dataset size, and data-augmentation techniques) can greatly improve performance ( Section 5.2 ).

• For output classifiers, the loss-function design significantly impacts optimization stability. We found that gradually transitioning from direct value-head probabilities to cumulative-maximum probabilities helped achieve stable training while maintaining streaming performance.

• Classification thresholds should be tuned to minimize impact on benign queries while maximizing robustness to jailbreak attempts.

Developing constitutions: • Refining the harmless constitution reduces false-positive rates. The clarity and comprehensiveness of harmless categories directly impacts the quality of synthetic training data and prevents classifiers from refusing benign queries.

• Regularly update the constitution based on observed weaknesses from evaluation results.

These guidelines reflect our experience developing constitutional classifiers, though we note that not all have been rigorously validated through controlled experiments. We hope these observations prove useful for others working on similar systems.

### Appendix B Example constitution

In Section 5.2 , we trained constitutional classifiers on a constitution that tries to identify harmful chemical-weapons information. Specifically, we use the constitution shown in Table 3 as the definition of “harmful” content for our synthetic-data-generation pipeline described in Section 3.2 . We also found that adding a specific “harmless” constitution that explicitly delineates examples of categories to consider harmless can significantly improve classifier performance (more details in Section E.6 ); we show this “harmless” constitution in Table 4 .

### Appendix C Simplified models of uplift

We now illustrate how our robustness results can be used to translate to real-world uplift. We use an illustrative example that models complex scientific processes that require N N sequential steps. Our model makes three key assumptions:

• Success requires completing all of the steps in the process.

• Each step has an independent and identically-distributed success probability. We assume a 95% success rate when the model provides similar detail to that of a helpful-only model, 50% with half of the detail of that of a helpful-only model, and 5% otherwise.

• The probability of successfully jailbreaking the model at each step is independent and identically distributed, and can be informed using red-teaming results. Based on our top red teamer’s performance across all submitted reports, here, we conservatively estimate that attackers using our safeguarded system have a 60% chance of obtaining detailed information and a 40% chance of obtaining partial information per step. This estimate likely significantly overstates attacker capabilities for three reasons: (i) red teamers can iterate their responses with feedback on the questions that contained at least half the detail of a helpful-only model, (ii) the rubric-grading process contained imperfections, and (iii) these results reflect only our best red teamer’s performance out of over one-hundred participants.