Constitutional Classifiers: Defending against Universal Jailbreaks across Thousands of Hours of Red Teaming — part 1/6

Cleaned-text code points [0, 15424). Bibliography omitted here; source.md includes it.

# Constitutional Classifiers: Defending against Universal Jailbreaks across Thousands of Hours of Red Teaming

###### Abstract

Large language models (LLMs) are vulnerable to universal jailbreaks—prompting strategies that systematically bypass model safeguards and enable users to carry out harmful processes that require many model interactions, like manufacturing illegal substances at scale. To defend against these attacks, we introduce Constitutional Classifiers : safeguards trained on synthetic data, generated by prompting LLMs with natural language rules (i.e., a constitution) specifying permitted and restricted content. In over 3,000 estimated hours of red teaming, no red teamer found a universal jailbreak that could extract information from an early classifier-guarded LLM at a similar level of detail to an unguarded model across most target queries. On automated evaluations, enhanced classifiers demonstrated robust defense against held-out domain-specific jailbreaks. These classifiers also maintain deployment viability, with an absolute 0.38% increase in production-traffic refusals and a 23.7% inference overhead. Our work demonstrates that defending against universal jailbreaks while maintaining practical deployment viability is tractable.

### 1 Introduction

Large language model (LLM) safety mechanisms can be circumvented by “jailbreaks” that elicit harmful information from models ( Shen et al., 2023 ; Liu et al., 2023 ; Qi et al., 2024 ; Andriushchenko et al., 2024 ; Anil et al., 2024 ; Hughes et al., 2024 ) . Such jailbreaks become more concerning as the chemical, biological, radiological, or nuclear (CBRN) capabilities of LLMs increase ( Anthropic, 2023a ; OpenAI, 2023 ; Li et al., 2024 ) . 2 2 2 This work was conducted as part of Anthropic’s Responsible Scaling Policy commitments to proactively mitigate misuse risks from increasingly capable language models.

To mitigate CBRN-related misuse risks, we focus on defending against universal jailbreak strategies: attacks that reliably extract detailed harmful information across the vast majority of queries in a domain, such as the “Do Anything Now” ( Shen et al., 2023 ) and “God-Mode” ( Pliny, 2025 ) attacks. Such universal jailbreaks are particularly concerning as they could allow non-experts to execute complex scientific processes that they otherwise could not have. Moreover, our defenses must be practically viable for deployment and flexible enough to adapt to evolving threat models.

To achieve these goals, we augment LLMs with classifiers that monitor model inputs and outputs to block potentially harmful content ( Figure 1 a). While individual classifiers may not achieve perfect robustness in isolation, they work together as complementary defensive elements in a “swiss-cheese” model ( Reason, 1990 ) , creating multiple layers of protection alongside the guarded language model. This simple approach is highly effective: in over 3,000 hours of human red teaming on a classifier-guarded system, we observed no successful universal jailbreaks in our target CBRN domain.

In particular, we introduce Constitutional Classifiers , a framework that trains classifier safeguards using explicit constitutional rules (§ 3 ). Our approach is centered on a constitution that delineates categories of permissible and restricted content ( Figure 1 b), which guides the generation of synthetic training examples ( Figure 1 c). This allows us to rapidly adapt to new threat models through constitution updates, including those related to model misalignment ( Greenblatt et al., 2023 ) . To enhance performance, we also employ extensive data augmentation and leverage pool sets of benign data. 3 3 3 Note we do not use production data for our benign pool, instead collecting data from external contractors. Critically, our output classifiers support streaming prediction: they assess the potential harmfulness of the complete model output at each token without requiring the full output to be generated. This enables real-time intervention—if harmful content is detected at any point, we can immediately halt generation, preserving both safety and user experience.

To evaluate the robustness of our approach, we conducted extensive human red teaming on prototype classifiers fine-tuned from Claude 3.5 Sonnet ( Section 4 ). 4 4 4 This preliminary system prioritized achieving high robustness, resulting in substantially elevated refusal rates compared to later implementations. We invited 405 participants, including experienced red teamers, to a bug-bounty program through HackerOne and offered substantial monetary incentives (up to $15K USD) for the discovery of universal jailbreaks. We tasked red teamers with obtaining answers to ten harmful CBRN queries, tying compensation to their success rate. Despite accumulating over 3,000 estimated hours of red teaming, 5 5 5 We estimate the red-teaming effort by analyzing usage logs. Our mean estimate is 4,720 hours, with 90% confidence intervals of [3,242, 7,417]. These confidence intervals are computed using a calibrated bootstrap procedure ( DiCiccio and Efron, 1996 ) , using participant self-reports for calibration. no red-teaming report succesfully answered all ten queries at half the level of detail of an unrestricted model, 6 6 6 One report initially appeared to achieve a universal jailbreak, but further investigation revealed this resulted from a deployment infrastructure flaw rather than classifier vulnerabilities. An implementation error allowed participants to receive up to 128 tokens after the output classifier had flagged harmful content. This finding emphasizes the importance of red teaming both classifiers and their deployment infrastructure. though red teamers did extract some harmful information. Under more-stringent criteria, requiring jailbreaks to provide a similar level of detail to an unrestricted model, no red teamer answered more than six of the ten questions. These results suggest that our red teamers did not find a universal jailbreak that consistently extracts information that is similarly detailed to a model without safeguards.

Following this, we establish the deployment viability and flexibility of our approach by training improved Claude 3.5 Haiku classifiers with a new constitution designed to block chemical-weapons information ( Section 5 ). On automated evaluations that include novel jailbreak attempts, we find that these classifiers substantially improve jailbreak robustness while preserving deployment viability. Specifically, our classifier-guarded system refuses over 95% of held-out jailbreaking attempts, compared to only 14% without classifiers. This improvement comes with limited costs: a 0.38% absolute increase in refusal rates on production Claude.ai traffic and a 23.7% inference overhead.

Overall, our work demonstrates that defending against universal jailbreaks while maintaining practical deployment viability is tractable.

### 2 Problem Formulation: Robust and Practically Viable Defenses Against Universal Jailbreaks

We now motivate our problem formulation: developing practically viable safeguards to defend against universal jailbreaks.

Desideratum 1: robustness to universal jailbreaks. Our goal is to develop safeguards that enable the wide deployment of LLMs with advanced scientific capabilities. Although such systems have beneficial uses, the dual-use nature of these capabilities raises important concerns. One particular concern, as highlighted in Anthropic (2023a) and OpenAI (2023) , is the potential for language models to give non-experts access to dangerous CBRN information.

Several threat models identify AI systems’ potential to uplift non-expert actors to expert-level capabilities as a core mechanism by which AI could lead to real-world harm in these domains ( Rose et al., 2024 ) . Such uplift could allow malicious actors to execute complex scientific and technical processes that would otherwise be beyond their capabilities or resources. For significant uplift, we believe the following conditions must be met. First, non-experts must be able to reliably obtain accurate information—they typically lack the expertise to verify scientific claims themselves. Second, the language model must successfully assist with the vast majority of distinct queries where the threat actor requires guidance. Third, the LLM must provide highly detailed, correct, and specific information rather than general knowledge.

Given these insights, we focus on mitigating universal jailbreaks, which we define as follows: a universal jailbreak or a universal jailbreak strategy is a (potentially automated) prompting approach that reliably bypasses LLM safeguards on the vast majority of queries in a specific domain, leading the system to reveal highly-detailed and specific harmful information. In other words, universal jailbreaks effectively convert models into variants without any safeguards .

We believe that preventing universal jailbreaks of LLMs would significantly reduce real-world CBRN risks once these risks become substantial. This is because universal jailbreaks enable precisely the type of non-expert uplift that poses the greatest concern: (i) they are reliable, allowing non-experts who cannot independently verify scientific information to consistently obtain accurate guidance; (ii) they work across the majority of queries in a domain, enabling assistance with the numerous steps involved in complex scientific processes; and (iii) they elicit detailed, specific information rather than general knowledge, providing the level of instruction needed to execute technical procedures.

Desideratum 2: practical deployment viability. Safeguards must remain viable for practical deployment. This means that the system must (a) maintain reasonable inference overhead and latency to be cost-effective in production, (b) preserve time-to-first-token and streaming capabilities for user experience, and (c) keep false-positive rates low to avoid frustrating legitimate users.

Desideratum 3: flexibility. Safeguards should be flexible enough to adapt to evolving domains of potential harm. Similarly, the system must be able to incorporate defenses against novel attack patterns as adversaries develop increasingly sophisticated techniques.

### 3 Constitutional Classifiers

To develop robust and practically viable defenses against universal jailbreaks, 7 7 7 We outline our overall approach here but omit certain details to preserve system security. we train classifier safeguards on constitution-guided synthetic data to detect and block jailbreaks. This constitutional approach provides significant flexibility in defining harmful and permissible content categories, allowing us to not only update safeguards to account for new and evolving threat models, but also generate vast amounts of training data. To achieve high robustness while minimizing false positives, we employ data-augmentation techniques and leverage large pools of benign inputs and outputs.

In particular, we employ a dual-classifier defense system consisting of input-only and output-only classifiers. The streaming output-only classifier filters the guarded model’s outputs and mitigates many prevalent universal jailbreak strategies ( Kim et al., 2024 ) , including many-shot jailbreaking ( Anil et al., 2024 ) and GCG ( Zou et al., 2023 ) . The input classifier can detect and block attempts to circumvent the output classifier.

#### 3.1 Specifying a constitution

Our approach uses a constitution (i.e., a set of natural-language rules) to define categories of permitted and restricted content (see Table 1 for an example). In general, specifying permitted content categories is essential for limiting over-refusals (see Section E.5 ).

The constitutional approach offers key advantages when developing robust safeguards. Because the constitution is formatted as natural language, it can be adapted and refined to address new threat models by updating the constitution’s categories of content. The labeled datasets produced by this method can be used to train different classes of safeguards (e.g., such as linear probes ( Alain and Bengio, 2016 ) ). Additionally, explicitly delineating both harmful and harmless categories produces more-nuanced training data, which helps classifier learn appropriate classification boundaries. The classifiers’ intended behavior is also interpretable by examining the constitution categories.

#### 3.2 Generating synthetic data

Our classifiers are fine-tuned LLMs, which offer several advantages over prompted classifiers: not only does fine-tuning let us systematically address vulnerabilities in our system, 8 8 8 We initially evaluated prompted classifiers through red-teaming partnerships with external organizations and individuals. but by specializing the LLMs for classification tasks, fine-tuning can also significantly reduce computational overhead.

Stage I: constitution-based synthetic-data generation. Given a constitution, e use a helpful-only model 9 9 9 Helpful-only language models are optimized for helpfulness without harmlessness optimization, which makes them particularly suitable for generating unrestricted responses to potentially harmful queries. to generate queries that span the constitutional categories, similarly to Anthropic (2024a) . We then use the helpful-only model to generate harmful and harmless outputs for each synthetically generated input. Finally, we use a prompted language model to filter out any inadvertent refusals from the generated inputs and outputs.

Stage II: data augmentation. To improve the diversity and coverage of the synthetic data, we apply extensive data augmentations. These “transformations” preserve semantic content while expanding the diversity of inputs and outputs, and they span multiple linguistic dimensions, incorporating techniques such as translation between languages, LLM-based paraphrasing, system-prompt variations and different jailbreaking techniques, as well as compositional combinations thereof.

#### 3.3 Automated red teaming

We generate additional training data using automated red teaming ( Perez et al., 2022 , ART;) . We prompt a helpful-only language model with descriptions of known jailbreaking techniques and ask the model to generate new attacks. We use the synthetic data from Section 3.2 as a source of forbidden queries for the model to use for red teaming. Furthermore, we allow our ART method to generate long-context and multi-turn attacks by prompting the model to generate an attack template that is then populated using additional calls to the language model. See Appendix F for further details.

#### 3.4 Training classifiers

Input classifier. We frame the input-classification task as a next-token-prediction problem and fine-tune a language model to perform this task ( Section E.1 ). Specifically, we wrap the entire input using a template that describes the task (see Section E.7 for complete prompt templates used in both classifiers). We use the data sources described in Section 3.2 and Section 3.3 as training data for our classifiers. Additionally, we train the classifier to block inputs that attempt to establish coded communication or otherwise obfuscate the output such that it cannot be understood without seeing the input. We also include pool sets of harmless queries to help minimize false-positive rates for legitimate inputs.