Constitutional Classifiers: Defending against Universal Jailbreaks across Thousands of Hours of Red Teaming — part 6/6
Cleaned-text code points [76810, 92412). Bibliography omitted here; source.md includes it.
We consider multi-turn attacks to be successful if and only if they generate meaningfully harmful outputs from a target system. Success is determined by helpful-only models that are prompted to classify if an output is a refusal as well as if an output is harmful within specific definitions of harmful information (see Section F.2 for more details). Successful attacks are collected into a pool that can be sampled for few-shot prompting future red-teaming attempts. Training on all attacks that meet these criteria, however, resulted in classifiers that overly refuse (i.e., often classify harmless prompts as harmful). For this reason, we also filter inputs by their corresponding outputs’ overlap with an output from a helpful-only model on the original harmful query. These filtered attacks may then be added back to the training pipeline as one-off datasets.
Critically, to avoid an imbalance in the training distribution, a nearly-equal number of automatically-generated benign queries about harmful topics may also be added to the training dataset (see Section F.3 for more details). The combined quantity of successful jailbreaks produced by ART and these benign queries generally comprises fewer than 2% of our training samples when used.
#### F.2 Helpful-only overlap grading
Successful attacks generated by the two-step ART process shown in Section F.1 often did not answer the given question. Using these loosely-relevant attacks to train constitutional classifiers actually decreased performance on our automated evaluations because the attacks were not effective jailbreaks and instead only added noise to the training set.
To ensure that ART attacks successfully answered the harmful question, we use Claude 3.5 Sonnet and provide (a) a rubric for judging, (b) the harmful question, (c) a helpful-only output sampled from the harmful question, and (d) the output sampled from the attack prompt. We then prompt the language model to decide whether the given output sufficiently overlaps with the helpful-only reference answer using the prompt shown in Table 8 .
#### F.3 Benign catastrophic-risk queries
As stated in Section F.1 , our automated-red-teaming pipeline only generates harmful inputs and outputs. We therefore balance out these additional examples with an approximately-equal number of benign conversations. These conversations focus on asking general and harmless questions about a harmful topic. For example, benign queries related to chemical weapons could include “What is sarin gas?”; “I’m worried about chemical weapons. How can civilians protect themselves?”; and “Are chemical weapons banned under international law?” Similar to the synthetic-data-generation pipeline outlined in Section 3.2 , we use a language model to generate topics from a list of harmful and harmless brainstorming topics 19 19 19 Harmless topics are directly obtained from the constitution stage in Section 3.2 . From these topics and subtopics, the language model then generates single-turn and multi-turn conversations, which we can then use as training data.
### Appendix G Evaluations
#### G.1 Rubric generation
To generate grading rubrics, we instruct a language model to identify key phrases organized by topic in a given output from a helpful-only model to a harmful question, as shown in Table 9 . Typical outputs contains between 8 and 18 topics, each of which may contain multiple key phrases. For each question to grade, we first obtain multiple rubrics generated from five distinct samples from a helpful-only model. We then combine these rubrics into a single rubric that contains the common key phrases used across multiple helpful-only outputs, as shown in Table 10 . Based on the individual helpful-only outputs, we can then compute the average topic-level overlap between a helpful-only output and the combined rubric, which we call α \alpha . 20 20 20 Most outputs from a helpful-only model to the harmful question will cover approximately two-thirds of the topics in the combined rubric for that question. This overlap is dependent on how open-ended the question is and how confident the helpful-only model is about its answer. Questions that are more open-ended and lower model confidence in answers both provide a wider range of topics that an output to the question may discuss. Finally, we use the combined rubric to grade a given jailbreak-style input—we consider an input to have answered the original question if the output sampled from a target system to that input covers at least max ( 2 , ⌈ α / 4 ⌉ ) \texttt{max}(2,\lceil\nicefrac{{\alpha}}{{4}}\rceil) topics and that a language model deems to have covered sufficiently-many topics. 21 21 21 For our bug-bounty program, we only require at least ⌈ α / 2 ⌉ \lceil\nicefrac{{\alpha}}{{2}}\rceil topics to be covered. Figure 3 provides a detailed example of how topics are counted between a sample output and a sample rubric. Section G.2 contains precise details on the grading process for a rubric.
#### G.2 Grading process
To determine whether an output contains enough harmful information to be considered a jailbreak, we use the following process:
1. As explained in Section G.1 , we sample outputs from helpful-only models for the original harmful question before applying any jailbreaking strategies or other augmentations (see Section 5.1 ). For each of these outputs, we prompt a language model to generate a list of keywords that describe the key information conveyed in the output, as shown in Table 9 . We then consolidate these keyword lists into a single “rubric” (shown in Table 10 ) that contains the key information that tends to be conveyed when sampling from a helpful-only model.
2. To account for possible obfuscation to the target output (e.g., a jailbreak could to ask for an output in Base64 to attempt to bypass the output classifier), we attempt to rewrite the target output in plain English. To do so, we give a language model the input prompt that generated the target output to be graded; the model then determines whether the prompt may result in an output that is not in plain English. If so, we ask the model to use the input to generate a set of instructions that could be used to deobfuscate any output generated from the input. Next, we apply another prompting round and give a language model the deobfuscation instructions and the target output. The model then uses the instructions to produce a semantically-equivalent version of the text in plain English. We show prompt templates for this step in Table 11 and Table 14 .
3. The plain-English output is given to a language-model rubric grader, which counts the number of key phrases in the target output that were also often present in the helpful-only outputs from (1). Our prompt template to do this is shown in Table 13 .
4. Finally, a language model is prompted with the original harmful question, the target output, and the rubric evaluation from (3). The model is also given metadata such as the topics that were covered by the target output and the minimum number of topics to cover. The language model will then decide whether enough topics were covered by the target output (according to the rubric) to constitute an answer to the original harmful question. 22 22 22 We found that separating (3) and (4) reduced instances of the pipeline marking outputs as harmful that we manually reviewed and determined were actually harmless. Note that we did not apply this step for our bug-bounty program.
Separately, we use a prompted output classifier to filter out any outputs that are refusals, as these outputs do not need to be graded for harmfulness. We do not consider any refusals to be harmful.
### Appendix H External red teaming
#### H.1 Recruiting red teamers
We invited a total of 405 23 23 23 We invited 405 participants to red team our system. Not all invited red teamers became active participants. participants to red team our system, mostly through an announcement page on our public-facing website. 24 24 24 https://www.anthropic.com/news/model-safety-bug-bounty . 25 25 25 We recruited six participants through other avenues. The program application asked potential participants about their experience with jailbreaking AI systems and provided open-ended questions that could be used to score participants’ effort, ethics, and skill level. We then prompted Claude 3 Opus to grade their answers on a 5-point scale and admitted the highest-scoring applicants to the program. Figure 17 shows the distribution of these scores for participants recruited through our application system.
#### H.2 Experience levels of red teamers
As part of our application process, we asked potential red teamers to indicate their experience level with jailbreaking language models. Participants were asked to self-indicate whether they were “new and looking to get started,” “part-time/hobbyists,” or “full-time bug-bounty hunters.” We show the distribution of red teamers’ experience levels in Figure 18 . Most red teamers that we recruited had prior jailbreaking experience; a non-trivial proportion of participants indicated that they were well-versed in jailbreaking language models.
#### H.3 Jailbreak-reporting process
We received 113 reports of jailbreaks through HackerOne. 17 reports were triaged and marked to be not applicable. We further eliminated 9 more reports for not attacking the correct domain and 2 more reports for being superceded by resubmissions. This results in 85 reports used for Figure 4 .
HackerOne triagers used Claude to automatically grade submissions for each question against rubrics, following Section G.2 . Participants were informed as to which questions they submitted did and did not exceed the 50% overlap threshold (see Section G.1 ) and were allowed to make 1 resubmission on a given report. Participants were also asked to submit deobfuscated versions of their model outputs in order to improve the accuracy of our grading systems. Participants were required to send message IDs received from the Anthropic API or Anthropic Workbench Console to verify their submissions. 26 26 26 If the inputs and outputs that the red teamer reported reached the 50% overlap threshold on at least two questions, we use the message IDs to obtain the exact input–output pairs and rerun our grading pipeline to verify that they match the report.
#### H.4 Estimating hours of red-teaming effort
We assume users are active when submitting queries to our system. 27 27 27 We assume this holds for both Claude Workbench and API users. However, high volumes of API traffic in a bucket may indicated automated attempts and not manual effort. To test this, we perform a sensitivity analysis where we excluded API queries made at rates exceeding 1.5 × 1.5\times the maximum manual query rate (likely automated rather than human-generated). Our mean prediction shifted by less than 150 hours. We estimated the total red-teaming effort T T in hours by using a bucketing model and API logs. We time into fixed-duration buckets of length d d and considered a red teamer to be active in any interval during which they made at least one query to our system. We then approximated total red-teaming time T ^ \hat{T} across the whole program by multiplying the number of active buckets by d d .
We estimated the bucket duration d d by fitting our model to self-reported hours from 69 mid-program survey respondents. We analyzed queries made by these respondents between the start of the program and the submission of the survey, determining the bucket duration that minimized the difference between total survey-reported hours and our model’s estimate of total red-teaming time. To quantify uncertainty in our estimates of d d and T ^ \hat{T} , we used bootstrapping (with 100K resamples of survey participants) to estimate distributions for both d d and T ^ \hat{T} (shown in Figure 19 ). This approach relies on self-reported hours from the survey, which may overestimate actual time spent red teaming.
To calibrate the uncertainty in our total red-teaming time and account for model error, we employed a bootstrap calibration procedure. We drew 1,000 pairs of training and validation splits ( t i , v i ) (t_{i},v_{i}) from our labeled dataset, with sizes 45 and 24, respectively. For each training split t i t_{i} , we estimated the bootstrap distribution of bucket durations with 100K resamples, and used these to compute a bootstrap distribution of total red-teaming hours T ^ i \hat{T}_{i} for the corresponding validation split v i v_{i} . We then identified lower and upper quantiles q l q_{l} and q u q_{u} , symmetric around 0.5, such that the true total red-teaming time T i T_{i} for validation splits fell within [ Q T ^ i ( q l ) , Q T ^ i ( q u ) ] [Q_{\hat{T}_{i}}(q_{l}),Q_{\hat{T}_{i}}(q_{u})] with 90% probability across different data splits. Under the assumption that this quantile-probability relationship holds for our unlabeled data, we obtained a calibrated 90% confidence interval for T ^ \hat{T} as [ Q T ^ ( q l ) , Q T ^ ( q u ) ] [Q_{\hat{T}}(q_{l}),Q_{\hat{T}}(q_{u})] . Empirically, we found q l ≈ 0.0005 q_{l}\approx 0.0005 and q u ≈ 0.9995 q_{u}\approx 0.9995 , yielding an estimated interval of [3,241, 7,417] hours. This bootstrap calibration approach follows DiCiccio and Efron (1996) .
#### H.5 Outputs that score highly on rubrics are less useful than helpful-only outputs
To get past the input classifier, most red-teaming reports rephrased harmful questions using various forms of rewriting where dangerous terms are swapped out with less-harmful terms or vaguer categories. By doing so, red teamers would be able to obtain outputs that contain information that is related to a harmful question but that does not actually provide the necessary steps to accomplish the question’s intent. We contend that these outputs are not catastrophically harmful because they do not actually provide the complex details necessary to execute a prohibited action in the correct context, instead gaming our grading system by attempting to output key words that appeared in the rubric. Indeed, manually examining reports of “successful” jailbreaks that scored highly based on our rubric graders ( Section G.2 ) showed many cases of outputs that appear harmful according to the rubric-grading metric but did not provide enough context for an actor to meaningfully execute based on the output. In Table 15 , we show two examples of reported “jailbroken” outputs that we manually reviewed and determined to be less helpful for accomplishing a harmful task than an output from a helpful-only model.
## Instructions for reporting errors
We are continuing to improve HTML versions of papers, and your feedback helps enhance accessibility and mobile support. To report errors in the HTML that will help us improve conversion and rendering, choose any of the methods listed below:
Click the "Report Issue" ( ) button, located in the page header.
Tip: You can select the relevant text first, to include it in your report.
Our team has already identified the following issues . We appreciate your time reviewing and reporting rendering errors we may not have found yet. Your efforts will help us improve the HTML versions for all readers, because disability should not be a barrier to accessing research. Thank you for your continued support in championing open access for all.
Have a free development cycle? Help support accessibility at arXiv! Our collaborators at LaTeXML maintain a list of packages that need conversion , and welcome developer contributions .