Constitutional Classifiers: Defending against Universal Jailbreaks across Thousands of Hours of Red Teaming — part 5/6

Cleaned-text code points [61488, 76810). Bibliography omitted here; source.md includes it.

We use this model as an illustrative tool that makes several key simplifying assumptions, rather than as an accurate model of scientific processes. First, it treats steps as independent, whereas real scientific processes typically involve complex dependencies and feedback loops. Indeed, in real processes, non-experts are likely able to recover from some errors or incorrect information by asking subsequent queries. Second, the discrete success probabilities (95%, 50%, and 5%) are coarse approximations of a continuous spectrum of partial success outcomes. Third, the assumption of uniform difficulty across steps overlooks the inherent variation in complexity between different steps in a process. Finally, our estimates of success rates derived from red-teaming exercises may not generalize to real-world adversaries with different capabilities and resources. Despite these limitations, the model demonstrates how reduced model capabilities can impact multi-step scientific processes.

Under this model, we observe an exponential decay in uplift as the number of process steps increases. For a 50-step process, under these conservative assumptions, constitutional classifiers reduce the uplift relative to a helpful-only model by five orders of magnitude, though we emphasise that this is an illustrative example that uses several simplifications.

### Appendix D System details

#### D.1 Bug-bounty system

As stated in Section 4 , we performed human red teaming on a prototype system containing classifier safeguards. These classifiers were trained to identify harmful content according to a different constitution from the one used for automated evaluations in Section 5.2 , though it still seeks to identify dangerous scientific knowledge related to weapons of mass destruction.

For the primary results in Section 4 , we initialized input and output classifiers from Claude 3.5 Sonnet. We also trained an improved version of the classifiers that were initialized from Claude 3.5 Haiku. This version of the classifiers achieved the same level of robustness with lower over-refusal rates as the original system, as stated in Section 4.3 . For both of these classifiers, we included data from automated red teaming, which we found improved classifier performance. We use these classifiers to guard Claude 3.5 Sonnet.

#### D.2 Automated-evaluations system

Here, we provide additional details on the classifiers used in Section 5.2 . We train these input and output classifiers to identify dangerous requests/information related to chemical weapons . Specifically, we train these classifiers on the constitution from Appendix B , which specifies that the classifiers are only intended to identify specific chemical-weapons-related information. An important distinction of the task is that our classifiers were not trained to identify harmful information that is not related to chemical weapons (e.g., planning a bank robbery). Compared to the constitutional classifiers used in the bug-bounty system ( Section D.1 ), we improved these classifiers by using a new constitution for synthetic-data generation, explicitly including a harmless constitution, choosing better hyperparameters during fine-tuning, increasing the amount of training data, and scaling the number of transformations used for data augmentation.

Furthermore, as stated in Section 5.2 , these input and output classifiers are both initialized from the pretraining-only version of Claude 3.5 Haiku (we use the pretraining-only checkpoint rather than the production Claude 3.5 Haiku model for the reasons shown in Section E.4 ). We also do not include data from automated red teaming when training these classifiers because our preliminary experiments showed that the “benign” examples from automated red teaming were too similar to the harmful categories specified in the constitution and thus made the training data more noisy. We use these classifiers to guard Claude 3.5 Sonnet (New).

### Appendix E Constitutional classifiers

#### E.1 Classifier types

In Section 3.4 , we described our formulation of input classification as a next-token-prediction task and output classification using value heads. To analyze these choices, we train input and output classifiers using next-token prediction and value-head setups.

As shown in Figure 9 , we find that next-token prediction improves performance over value heads for both input and output classifiers, although the improvement is more significant for the input classifier. We hypothesize that task similarity, overfitting potential, and in-context task specification play a role in this phenomenon:

1. Next-token prediction more-closely aligns to the pre-training language-modeling task. By framing the classification task as a language-generation problem, the model can leverage its existing knowledge about language patterns, context, and semantics. The value-head classifier, on the other hand, may be unable to utilize this existing knowledge as effectively.

2. Value-head classifiers introduce a new set of parameters that need to be trained. This additional layer may be prone to overfitting, especially in settings with limited training data. Next-token prediction, on the other hand, uses the model’s existing parameters; as these have been trained on a diverse large-scale dataset during pre-training, they may generalize better.

3. Next-token prediction provides additional context on the classification task within its prompt (i.e., it frames the intended classification task and the possible classes). The model can use this context to improve the accuracy of its predictions.

In light of these results, we choose next-token prediction for our input classifier. For our output classifier, however, we choose the value-head setup despite its lower performance because of the simplified implementation of allowing token streaming for the end-to-end system.

#### E.2 False-positive rates on domain-specific datasets

In Section 5.2 , we found that our classifiers achieved significant gains in robustness without refusing a significant amount of user traffic. Our classifiers’ performance differs, however, when analyzing specialized scientific content. While constitutional classifiers achieve a low false-positive rate of 1.50% on MMLU–Chemistry questions, we observe notably-higher FPRs of 26.05% on GPQA–Chemistry questions. This increase in false-positive rates highlights a key challenge—accurately classifying complex technical content often requires sophisticated domain understanding that may exceed the capabilities of the classifier.

When we study how domain-specific false-positive rates change with respect to scaling the classifier, however, we see an encouraging trend. We find that when robustness is fixed, scaling up the classifier generally reduces the false-positive rate on domain-specific questions, as shown in Figure 10 . This suggests that the higher false-positive rates on challenging chemistry problems primarily reflect a capability limitation that can be addressed by scaling up the classifiers rather than a fundamental limitation of our approach. Additionally, as model capabilities continue to improve over time, these false-positive rates may become more and more modest.

#### E.3 Scaling few-shot prompted classifiers

Here, we evaluate how much performance can be achieved by scaling the number of few-shot examples given to a prompted Claude 3.5 Sonnet classifier. To do this, we first identify training examples from Section 3.2 that the zero-shot prompted classifier predicts incorrectly on. We then select a random subset of these training examples, filtering for examples that are fewer than 1K tokens in order to conform to context-length constraints when using many in-context examples. Finally, we construct few-shot prompts by alternating selected examples such that prompts contain an equal number of positive and negative examples.

As shown in Figure 11 , we find that adding more few-shot examples generally improves the performance of the prompted classifier. As more examples are added to the prompt, however, the context length of the classification task increases linearly. This means that it is not feasible to continue to scale the number of few-shot examples given to the prompted classifier. This few-shot approach, however, does not readily support token streaming and still does not match the performance of a smaller constitutional classifier (as shown in Section 5.2 ).

#### E.4 Initializing classifiers from finetuned models

In Section 5.2 , we evaluated classifiers that were initialized from base Claude 3.5 Haiku. An open question, however, is whether the classifiers should be initialized from base or fine-tuned language models. To test this, we trained classifiers using different initializations while keeping datasets and training hyperparameters fixed. We initialized the classifiers from base Claude 3.5 Haiku, Claude 3.5 Haiku with training to be helpful, honest, and harmless (HHH), and Claude 3.5 Haiku with additional training to only be helpful (even if it requires being harmful).

As shown in Figure 12 , initializing from the base model achieves the best performance for both input and output classifiers. Adding HHH training reduces performance, likely because the training process creates biases that conflict with our task-specific definition of harmful content (e.g., a model may be trained to consider all discussions of explosives as harmful, whereas our constitution makes a finer distinction to only consider chemical-weapons explosives). Further helpful-only training also degrades performance compared to the base model, likely for similar reasons. These results suggest that constitutional classifiers should be initialized from base models, as they provide the most-malleable starting point for learning the specific harm definition for the classification task.

#### E.5 Robustness drops without a harmless constitution

Here, we evaluate the significance of specifying a constitution of harmless topics. To do, we train our classifiers without the synthetic harmless data generated from the “harmless” constitution shown in Appendix B , but we keep a fixed pool of benign inputs and outputs to use as harmless training data.

Figure 13 shows that including the harmless-constitution data improves robustness by over 40%. This is likely because the harmless-constitution data allow the model better delineate harmless queries that we use to calibrate our classification thresholds. We thus see that specifying both harmful and harmless constitution categories is crucial for achieving high robustness with low false-positive rates. Additionally, preliminary experimentation showed that we could reduce false positives using a feedback loop where we retrain classifiers with added harmless constitution categories that a previous version of the classifier tended to over-refuse on. For this reason, we expect that further specification of harmless categories in addition to the ones from Appendix B may be able to continue to improve classifier performance.

#### E.6 Importance of categories in a harmless constitution

Section E.5 showed that adding synthetic data generated from a harmless constitution is crucial for allowing classifiers to have high robustness with low false-positive rates. Here, we examine whether specific types of harmless categories are especially useful for improving classifier performance. We separate the categories of our “harmless” constitution in Appendix B into three buckets: (1) harmful non-chemistry, (2) harmless chemistry, and (3) harmless non-chemistry. 18 18 18 Harmful chemistry topics would be the categories in the harmful constitution in Appendix B . We then ablate removing each of the buckets from the training set and retrain our classifiers (all training sets included the fixed set of benign inputs and outputs, following Section E.5 ).

Figure 14 shows that although there isn’t a specific bucket of categories that had the greatest effect on performance for both the input and output classifier, failing to include any of the aforementioend buckets clearly reduces classifier performance, especially for the output classifier. These results are expected, as providing proper training data that specifies what types of information are harmless (rather than only including a fixed set of benign inputs and outputs) helps the classifier understand where to draw the distinction between harmful and harmless data.

#### E.7 Prompt wrappers

As stated in Section 5.2 , we compare our constitutional classifiers against prompted classifiers as baselines. Additionally, our input classifier is trained using a prompt wrapper that frames its next-token-prediction task, as mentioned in Section 3.4 . We show the prompt wrapper that we use for our constitutional input classifier (and zero-shot prompted input classifier) in Table 5 . Table 6 shows the prompt wrapper that we use for our zero-shot prompted output classifier. In both cases, we attempt to block inputs and outputs that use coded communication or otherwise contain obfuscated outputs, following Section 3.4 .

#### E.8 Refusal rates on chemistry-related user queries

To better understand the extent to which our classifiers overly refuse on chemistry questions that are similar to what real users may ask a language model, we create a chemistry-specific subset of Wildchat ( Zhao et al., 2024 ) . To do so, we zero-shot prompt Claude 3.5 Sonnet to classify whether a given conversation is related to chemistry (see Table 7 for the prompt that we use), and we run this prompted classifier on all examples. We then filter for conversations that the prompted classifier predicted to be about chemistry with a confidence greater than or equal to 0.99. Finally, we randomly sample 500 single-turn and 500 multi-turn queries from this filtered set of conversations to create our final dataset comprising 1K examples.

We sample model outputs from Claude 3.5 Sonnet (new) and find that the model refused to answer 3.6% of the inputs. We then run our constitutional classifiers from Section 5.2 on these inputs and outputs, finding that the input and output classifiers blocked an additional 3.4% of queries that did not result in a model refusal. These results indicate that the additional false positives incurred by using constitutional classifiers are more noticable for chemistry-related prompts. We believe that this overhead is still relatively moderate, however, especially since Section E.2 shows that we can likely expect this overhead to decrease as models continue to improve.

### Appendix F Automated red teaming

#### F.1 Attack generation

The goal of automated red teaming (ART) is to generate long, multi-turn attacks that can jailbreak a given system. We do this by first generating an outline of an entire attack conversation then filling the outline with concrete instances of a conversation. The outlining step produces several stages (i.e., high-level descriptions) that outline a conversation, with each stage representing either an entire conversation turn or a fraction of a turn. To fill the outline, the red-teaming model resamples a filled version of each stage. This two-step process ensures that generated attacks are sufficiently long and complex.