Constitutional Classifiers: Defending against Universal Jailbreaks across Thousands of Hours of Red Teaming — part 3/6

Cleaned-text code points [31048, 46394). Bibliography omitted here; source.md includes it.

Limitations. Our red-teaming evaluation has several important limitations. Although our prototype system demonstrated the feasibility of universal jailbreak defenses, its prioritization of robustness led to impractically high refusal rates. Moreover, red teamers did identify some vulnerabilities, suggesting that further improvements to robustness can be made. Our red-teaming cohort also only represents a subset of potential adversaries, and new attack strategies will emerge as jailbreaking techniques evolve. Furthermore, red teamers received payouts as soon as they achieved half the detail of a helpful-only model—they could potentially find more-powerful attacks given sufficient incentives. Our rubric-based evaluation also had difficulty distinguishing between benign knowledge and actionable information for capabilities uplift, potentially introducing both false positives and negatives. Finally, our evaluation focused on a chatbot setting with preset queries, leaving open questions about classifier robustness in other deployment contexts.

#### 4.3 Comparative assessment of defensive components

To evaluate how each defensive component in our system contributes to model robustness, we analyze attack success rates across randomly sampled prompts from our red-teaming efforts. We assess system responses using rubric-based grading criteria with a half-as-detailed-as-helpful-only threshold to calculate attack success rates across different model configurations.

Constitutional classifiers provide substantially stronger safeguards than harmlessness training alone. Our helpful-only baseline model, which lacks safety training, exhibits a ∼ \sim 16% attack success rate (ASR), 13 13 13 This low ASR like stems from red teamers modifying question semantics to circumvent safeguards. as shown in Figure 5 . Although adding harmlessness training (HHH model) modestly reduces this to ∼ \sim 14%, incorporating constitutional classifiers yields dramatically stronger protection. Adding an input classifier reduces the attack success rate to just 2%, and combining both input and output classifiers further reduces it to 0.5%.

Improved constitutional classifiers. In the following section, we present an improved constitutional classifier method that achieves lower refusal rates and inference overhead than the system that was red teamed. When applied to this constitution, this methodology yields lower attack success rates ( Figure 5 ) compared to our red-teamed system while simultaneously maintaining an acceptable false-positive rate (approximately 1.74% on production Claude.ai traffic). Although this system uses the same methodology as described in Section 5 , it employs a different constitution. We note that since red teamers did not specifically target this system, these results may not be fully representative.

### 5 Constitutional classifiers are flexible and practically viable defenses

We now demonstrate the flexibility and deployment viability of Constitutional Classifiers. Specifically, we train classifiers with enhanced methodology to defend against potentially dangerous chemistry-related queries, using a constitution distinct from that of our previous experiments. Our results show that these classifiers offer strong robustness with limited inference overhead and over-refusals.

#### 5.1 Methodology

We fine-tune input and output classifiers using a chemical-weapons constitution (shown in Appendix B ). Both classifiers are created by fine-tuning Claude 3.5 Haiku and are used to guard Claude 3.5 Sonnet (New). We then assess this jailbreak defense using a suite of automated evaluations.

Enhanced classifier methodology. We implemented several improvements to reduce our system’s false positive rate. First, we substantially expanded our harmless constitution. We then recalibrated our classifier thresholds and expanded our data-augmentation techniques. We later show increasing both the scale of training data and the number of augmentations improves performance. Additionally, we reduced inference overhead by training classifiers using Claude 3.5 Haiku rather than Claude 3.5 Sonnet. Further details are shown in Section D.2 .

Jailbreak-attempts dataset. We construct a dataset of jailbreak attempts targeting ten pre-specified chemical-weapons-related queries. To generate attacks targeting these queries, we follow Doumbouya et al. (2024) and compose jailbreak primitives , which we derive from three primary sources: existing jailbreaking strategies documented in the literature ( Mazeika et al., 2024 ; Anil et al., 2024 ; Souly et al., 2024 ) , novel attack vectors identified through internal red teaming of prototype systems, and text-transformation techniques including cryptographic ciphers, cross-lingual translations, and LLM-driven semantic reformulation. To ensure dataset quality, we implement a filtering mechanism that eliminates prompts that fail to elicit harmful responses from a helpful-only model, evaluated using LLMs. Given that certain text modifications overlap with those used in training-data augmentation (detailed in Section 3.2 ), we additionally construct a held-out subset comprising 5,681 jailbreak attempts that uses primitives and transformations absent from the training augmentation pipeline. This held-out set includes some entirely novel jailbreaks, as well as jailbreak attempts specifically designed to circumvent classifier-guarded systems.

Grading model responses. To evaluate jailbreak success, we again use rubric grading. We use the same process as in Section 4 to generate rubrics. To evaluate candidate model outputs, we first convert potentially obfuscated outputs (e.g., Base64-encoded text) into plain English using a prompted LLM. We then evaluate preprocessed outputs against the rubrics using a language-model grader. For a successful jailbreak, the output must also not be flagged by an LLM prompted to flag refusals. Furthermore, the final LLM grader must confirm that the output answers the original harmful question, verifying the rubric evaluation shows coverage of at least 25% of topics present in helpful-only responses and at least two distinct harmful topics (see Section G.2 for complete details of the grading process).

Over-refusal datasets. We evaluate the refusal rate of our approach on a dataset of 5,000 conversations randomly sampled from Claude.ai production traffic, drawn from both Free and Pro users. To better assess false-positive rates in chemistry-related queries, we evaluate our approach on both everyday user questions and scientific content. Specifically, we examine a subset of Wildchat conversations containing benign chemistry-related user queries (identified using LLM-based classification). We also evaluate on technical chemistry questions from GPQA ( Rein et al., 2023 ) and MMLU ( Hendrycks et al., 2020 ) in Section E.2 .

Setting classification thresholds. To set classification thresholds, we use Wildchat ( Zhao et al., 2024 ) and OR-Bench ( Cui et al., 2024 ) . In particular, we sweep over possible threshold combinations for input and output classifiers and select the thresholds that maximize the true-positive rate on our jailbreaking dataset while keeping the full system’s false-positive rate (FPR) below the following limits: ≤ \leq 0.5% increased FPR (compared to an unguarded system) on multi-turn queries from Wildchat, ≤ \leq 5% FPR on single-turn queries from Wildchat, and ≤ \leq 5% increased FPR on OR-Bench.

Estimating inference costs. To estimate inference costs, we apply estimates of the costs per input and output token on a randomly-sampled set of queries from Claude.ai production traffic. For prompted models, we assume that large prompts have been cached (see Section A.2 for more details). We also assume efficient implementations of classifiers (e.g., using optimizations such as prompt caching).

#### 5.2 Results

We compare the performance of our classifier-guarded system against multiple baseline systems. The most-prevalent deployment scenario currently in use is an unguarded instance of Claude 3.5 Sonnet (New) relying solely on harmlessness training. We also use three variants of prompted classifier safeguards based on Claude 3.5 Sonnet: (i) zero-shot prompting; (ii) chain-of-thought prompting ( Wei et al., 2022 ) ; and (iii) 32-shot prompting using exemplars from our training dataset.

Constitutional classifiers substantially enhance robustness while preserving model utility. Our evaluation demonstrates that augmenting Claude 3.5 Sonnet (New) with constitutional input and output classifiers yields significant improvements in jailbreak resistance. Among all evaluated baselines, constitutional classifiers achieve the highest robustness, reducing the jailbreak success rate from 86% against a HHH model to less than 5% ( Figure 6 A). We also found that no single transformation could answer all ten queries, suggesting that these classifiers also protect against universal jailbreaks. These substantial gains incur limited costs—we observe a small increase in false-positive rates on production traffic ( Figure 6 B) and moderate computational overhead during inference ( Figure 6 C, see Section A.2 for details on overhead calculations). The increase in false-positive rates is more noticeable, however, for chemistry-related queries ( Section E.8 ).

Classifier robustness scales with model size, data quantity, and data augmentation. We train classifiers from model snapshots of five different sizes from the Claude 3 Sonnet series, fixing hyperparameters and training-data size 14 14 14 For these experiments, we excluded examples exceeding 100K tokens from training . We find that robustness consistently increases with model size and that larger models exhibit smaller generalization gaps for input classification (as shown in Figure 7 A). Moreover, by fine-tuning Claude 3.5 Haiku classifiers on subsets of training data, we observe substantial improvements in robustness when using larger training datasets ( Figure 7 B). Finally, we analyze the impact of data augmentation methods, demonstrating that incorporating additional transformations generally increases classifier robustness ( Figure 7 C).

### 6 Related Work

Classifier guards. Recent work also explores classifier-based approaches for safeguarding language models. Markov et al. (2023) identified the role of data quality and active learning in training moderation APIs and similarly used synthetic data. Inan et al. (2023) ; Chi et al. (2024) ; Rebedea et al. (2023) also developed performant classifier safeguards. Kim et al. (2024) argued that output filtering resolves many prevalent current jailbreaks. Wang et al. (2024) also studied classifier safeguards but in the narrow domain of bomb-making. Our work suggests that additional classifier fine-tuning would improve performance. The flexibility of this approach allowed us to quickly (without manual data collection) adapt our classifiers when we noticed vulnerabilities throughout development. Our work shows that classifier-based defenses can be made to obtain robustness to universal jailbreaks in the face of thousands of hours of red teaming.

Finetuning on red-teaming attacks. One popular approach for improving robustness is to red team LLMs for harmful behaviors—either manually ( Ouyang et al., 2022 ; Ganguli et al., 2022 ) , or automatically using LLMs ( Chao et al., 2024 ; Mehrabi et al., 2024 ; Samvelyan et al., 2024 , e.g.,) —then fine-tune LLMs to not exhibit those harmful behaviors. This approach is commonly used to train frontier LLMs such as Claude ( Anthropic, 2023b ; Anthropic, 2024b ) and Llama 3 ( Dubey et al., 2024 ) . It leaves models susceptible to a variety of universal jailbreaks, however, including many-shot jailbreaking ( Anil et al., 2024 ) and GCG ( Zou et al., 2023 ) . In our preliminary experiments, we found that fine-tuning did not reliably generalize well (e.g., from text to code settings) when attempting to train away harmful behaviors on successful red-teaming attacks. These limitations led us to explore classifier-based safeguards for defending against universal jailbreaks.

Model-internals approaches. Some LLM robustness approaches leverage access to internal model representations. A simple approach is using linear probes on model activations to detect harmful intent ( Alain and Bengio, 2016 ; Ousidhoum et al., 2021 ) . More advanced approaches fine-tune LLMs using model-internals-based loss functions, such as short-circuiting ( Zou et al., 2024 ) and latent adversarial training ( Casper et al., 2024 ) . We note that the datasets produced by our synthetic data generation pipelines can be used these approaches. Moreover, constitutional classifiers do not require modifications to the LLM fine-tuning procedure, which makes them more flexible and easier to deploy in practical settings. Finally, it is unclear whether these approaches alone can be used to match the level of universal jailbreak robustness achieved by classifier-based approaches.

Unlearning and data filtering. An orthogonal method for improving LLM robustness is to either “unlearn” the hazardous knowledge that exists within a model ( Li et al., 2024 ; Zhang et al., 2024 ) , or to prevent that information from being learned by the model using pre-training data filtering. Unfortunately, machine unlearning often fails to fully erase undesired knowledge ( Lynch et al., 2024 ; Shi et al., 2023 ) . Moreover, pre-training data filtering lacks the flexibility of our approach.

Robustness via scaling inference-time compute. Zaremba et al. (2024) explore how increased inference-time compute can enhance the adversarial robustness of reasoning models. While this approach shows promise, it incurs increased latency and potentially large inference overheads, depending on the extent of reasoning necessary for sufficient robustness. Our classifier approach offers a powerful complementary defense.

### 7 Conclusion

We present Constitutional Classifiers: safeguards trained on LLM-generated synthetic data using constitutions of natural-language rules of permitted and restricted content. Despite the simplicity of our approach, our experiments demonstrate that it provides substantial improvements in robustness. In extensive human red teaming of our prototype system, no red teamer discovered a universal jailbreak capable of consistently extracting information comparable to an unsafeguarded model. Through subsequent improvements, we achieved high robustness while minimizing both over-refusals and inference overhead. Moreover, the constitution-based approach provides significant flexibility in adapting to novel threats via updates to the constitution.

While these results are promising, common wisdom suggests that system vulnerabilities will likely emerge with continued testing. Responsibly deploying advanced AI models with scientific capabilities will thus require complementary defenses. These defenses may include jailbreak rapid response for vulnerability patching ( Peng et al., 2024 ) , as well as monitoring techniques to identify novel jailbreaks ( Hendrycks et al., 2021 ) . Nevertheless, we expect Constitutional Classifiers to play a crucial role in safely deploying capable AI systems, such as those requiring ASL-3 deployment standards.

### Acknowledgements