Rainbow Teaming: Open-Ended Generation of Diverse Adversarial Prompts — part 2/4

Cleaned-text code points [15461, 31088). Bibliography omitted here; source.md includes it.

Relying on a preference model rather than a score-based evaluator offers two advantages. First, LLMs prompted to perform pairwise comparisons have a higher agreement with humans than those performing single-answer grading ( Zheng et al., 2023 ) . This is particularly true in an optimisation context, which introduces the risk of reward hacking the evaluator. Second, the score of any numerical evaluator with a fixed scale can be maximised, at which point it is impossible to identify better candidate prompts, resulting in minimal updates in the archive. We present a preference model ablation supporting those claims in section E.4 .

While we describe Rainbow Teaming as using LLMs for all key steps, those can be substituted by other models or rule-based components in some domains (e.g., see section 6.1 ).

## 4 Rainbow Teaming for Safety

In this section, we evaluate the effectiveness of Rainbow Teaming in discovering adversarial prompts that trigger unsafe (i.e., harmful, offensive, or toxic) responses. We target various models that are fine-tuned with safety-aligned data and are among the safest publicly available LLMs, including Llama 2 and Llama 3 models.

• Features: We use two feature dimensions: Risk Category and Attack Style. Our risk categories expand upon the safety risk taxonomy in Inan et al. (2023) and cover common safety issues in LLMs, such as “Criminal Planning”, “Violence and Hate” or “Self-Harm”. Attack Style pertains to the method of eliciting unsafe responses, such as “Authority Manipulation” and “Role Play”. Together, these features enable a thorough evaluation of LLM safety, covering a broad spectrum of adversarial threats and tactics. The full list of categories for both features is provided in section H.1 .

• Mutation Operator: We use an instruction-tuned Llama 2 70B model, which applies two mutations to create the candidate prompt, as illustrated in fig. 2 . The first mutation targets the risk category, using a few-shot prompt template shared across all categories. The second mutation focuses on the attack style, employing few-shot prompts handcrafted for each individual category.

• Preference Model: We use an instruction-tuned Llama 2 70B model as a Judge to compare Target responses to two adversarial prompts. Its role is to determine which response is more unsafe, ignoring the prompts. The prompt corresponding to the winner’s response is the one we then keep in the archive. Details about the prompts used for the Judge are provided in appendix I .

To initialise the archive, we generate prompts from scratch according to risk categories without relying on a pre-existing seed dataset. These are then mutated with respect to attack style and used as a foundation for subsequent evolutionary iterations. We use two independent measures of Attack Success Rate (ASR) – the proportion of prompts in the archive that elicit unsafe responses. Specifically, we use two safety classifiers, namely GPT-4 ( OpenAI, 2023 ) with a specialised prompt and the Llama Guard safeguard model ( Inan et al., 2023 ) . GPT-4 acts as a binary classifier for whether a response is generally unsafe or not, independent of risk categories. In contrast, Llama Guard is prompted to perform per-category binary classification ( Inan et al., 2023 ) , considering an attack successful only if the resulting response violates the risk category it is assigned to. Neither of these metrics is explicitly optimised by Rainbow Teaming , but the probability of Llama Guard classifying a prompt as unsafe is the fitness score used to bias the selection of the prescribed feature descriptors for new candidates. Prompts for both evaluators are provided in appendix I . For all experiments, we report the mean and standard error over 3 3 independent runs.

We also measure inter-evaluator agreement on 100 pairs of prompts and responses. table 8 in section E.3 shows that human-human agreement (83%) is similar to human-AI agreement (81% for GPT-4 and 78% for Llama Guard) and GPT-4-Llama Guard agreement (79%), and is consistent with prior work ( Zheng et al., 2023 ) . We therefore use GPT-4 and Llama Guard as proxies for human evaluation.

### 4.1 Results

Main Results. fig. 4 presents the ASR of Rainbow Teaming when applied to the Llama 2-chat 7B ( Touvron et al., 2023 ) , Llama 3-Instruct 8B ( AI@Meta, 2024 ) , Mistral 7B ( Jiang et al., 2023 ) and Vicuna 7B v1.5 ( Chiang et al., 2023 ) models across 2000 iterations, using GPT-4 for evaluation. Rainbow Teaming is highly effective, generating a large collection of adversarial prompts against all models. The Llama models exhibit the highest robustness: following 2000 iterations, we obtain archives of 100 prompts with an approximate ASR of 92% against both variants. Mistral 7B and Vicuna 7B demonstrate a higher level of vulnerability with 98% of the adversarial prompts in Rainbow Teaming -generated archives being successful. These results are echoed by the ASR reported by Llama Guard in fig. 10 .

While fig. 4 showcases relatively small LLMs, Rainbow Teaming is equally effective against larger models. fig. 8 in section E.1 presents results of Rainbow Teaming targeting 7B, 13B, and 70B variants of Llama 2-chat model, achieving 90% or higher ASR across all model sizes .

We compare Rainbow Teaming to two baselines. The first baseline (No Stepping Stones) ignores past solutions in the archive and generates new prompts based on the risk category, before applying the attack style mutation, effectively repeating the process we use to initialise the Rainbow Teaming archive. The second baseline, (Same Cell Mutations) , is identical to Rainbow Teaming , except that it uses the parent prompt’s descriptor as the candidate prompt descriptor, i.e., it performs mutations within each archive cell independently. fig. 4 shows Rainbow Teaming outperforming both baselines, highlighting the value of stepping stones in one case and the significance of cross-category mutations in the other.

JailbreakBench Results. We also apply Rainbow Teaming towards eliciting specific harmful behaviours from the JailbreakBench ( Chao et al., 2024 ) dataset. Using the same attack styles, we generate 1000 prompts evenly spanning 100 harmful behaviours, with results presented in table 1 . We compare against two PAIR ( Chao et al., 2023 ) variants: one from Chao et al. (2024) , based on MiXtral, and another using the same mutator LLM as our Rainbow Teaming implementation, with N = 20 N=20 parallel streams generating a total of 2000 prompts. We classify jailbreaks using both the same classifier as Chao et al. (2024) and Llama Guard prompted with the harmful behaviours. For each prompt, we regenerate 4 responses and consider the prompt successful if any of the responses is classified as harmful. We believe this is representative of user interaction with LLMs, where they can prompt the model repeatedly in the hope of obtaining a different response. Compared to both PAIR variants, Rainbow Teaming discovers more jailbreaks across more behaviours, while also maintaining much higher prompt diversity.

Transfer of Adversarial Prompts. Understanding whether attacks transfer across models is important to assess the generality of the adversarial prompts, and whether they are intrinsically tied to the models they are optimised for. To evaluate transfer, we take the final prompts generated by Rainbow Teaming for each original target in Figure 4 and evaluate their ASR against other transfer targets .

table 2 presents the ASR on four different models using archives generated by Rainbow Teaming targeting each of these models. We show the ASR in grey when re-prompting targets using their own archive. On average, the ASR when transferring prompts is 50 % 50\% of the ASR against the original target, indicating that Rainbow Teaming discovers general prompts which apply to multiple models. However, the exact transfer rate is highly dependent upon the pairing of original and transfer targets. We find that prompts transfer better from safer to less safe models than in the opposite direction. That said, the highest transfer rate is from Vicuna 7B 1.5 to Mistral 7B, even though Vicuna is fine-tuned from a Llama 2 base. We also achieve up to 66% ASR on GPT-4o, indicating no significant difference between open and closed-source models.

Impact of the Similarity Filter. Because archive categories are not mutually exclusive, we run the risk of populating the archive with near identical prompts. This is useful for discovering a category-agnostic failure mode but comes at the cost of significant diversity loss in the archive. To mitigate the issue, we implement a parent-child similarity filter at the mutation stage, as described in section 3.2 . table 3 compares the performance of Rainbow Teaming with and without using this similarity filter. We also report archive self-BLEU ( Zhu et al., 2018 ) , BERTScore ( Zhang et al., 2020 ) , ROGUE-L ( Lin and Och, 2004 ) m and compression ratio ( Shaib et al., 2024 ) scores designed to measure the diversity of a whole dataset. Our results show that the similarity filter is an effective way of maintaining the linguistic diversity of the archive.

Additional results with different system prompts are provided in section E.2 . We include an ablation study in section E.4 to assess the role of the preference model. We discuss computational costs in appendix G .

## 5 Enhancing Robustness with Synthetic Data

Generating diverse, high-quality instruction-tuning datasets can be expensive, often requiring human annotations. Rainbow Teaming offers a low-cost alternative, generating diverse synthetic data that specifically targets the model’s vulnerabilities. In this section, we demonstrate the usefulness of Rainbow Teaming as a synthetic dataset generation method by applying it to improve the safety of LLMs. We find that training on our synthetically generated data improves robustness to adversarial prompts while retaining the general capabilities of the model.

We use Rainbow Teaming to generate 15 archives targeting the Llama 2-chat 7B model, yielding a total of 1500 adversarial prompts. We perform a 12/3 train-test split and use Llama 2-chat 70B with a handcrafted system prompt to generate safe refusal prompts for the train set. We then perform supervised fine-tuning (SFT) ( Wei et al., 2022 ) on this dataset and evaluate the ASR of the 300 held-out prompts before and after SFT. As shown in table 4 , we find that fine-tuning Llama 2-chat 7B on the synthetic dataset generated by Rainbow Teaming substantially reduces the attack success rate from 92% / 95% to 0.3% / 0.7% , as measured by GPT-4 and Llama Guard. Similarly, the ASR of PAIR ( Chao et al., 2023 ) on the JailbreakBench (JBB, Chao et al. (2024) ) behaviours drops from 14% to 0% (measured by Llama Guard, as in table 1 ). This demonstrates that additional SFT on Rainbow Teaming data also improves safety against out-of-distribution attacks. Crucially, SFT does not diminish the model’s general capabilities as measured on the GSM8K (8-shot, maj@1) ( Cobbe et al., 2021 ) and MMLU (5-shot) ( Hendrycks et al., 2021 ) benchmarks. 4 4 4 Touvron et al. (2023) report base model scores on these benchmarks while we report those of the chat model.

table 4 also reports the reward model scores ( Touvron et al., 2023 ) of the Llama 2-chat 7B model before and after SFT. We report safety and helpfulness scores on the Anthropic Harmless and Anthropic Helpful datasets ( Ganguli et al., 2022 ) respectively. We observe a 1.5 % 1.5\% safety score increase, despite the fact that Llama 2-chat models use the Anthropic Harmless dataset as a part of the reinforcement learning from human feedback (RLHF) pipeline ( Touvron et al., 2023 ) . This is accompanied by a 0.5 % 0.5\% drop in helpfulness, which we attribute to fine-tuning the model exclusively on the adversarial prompts produced by Rainbow Teaming . Mixing the adversarial data with helpfulness data would likely negate this effect, but we leave the study of adversarial fine-tuning strategies to future work.

To further investigate the robustness of the newly fine-tuned model, we reapply Rainbow Teaming to the Llama 2-chat 7B model after fine-tuning it on synthetic data generated by our method. As shown in fig. 5 , the new model is substantially more robust to our approach, with a final ASR of 39% (down from 92%) . We expect that performing multiple rounds of Rainbow Teaming , alternating between collecting synthetic data and adversarial fine-tuning, will further increase the model’s robustness to adversarial attacks. We show examples of archives at different iterations of Rainbow Teaming before and after SFT in fig. 13 .

## 6 Rainbow Teaming for Other Applications

### 6.1 Question Answering

We apply Rainbow Teaming to question answering, generating adversarial trivia questions — questions which the target model answers incorrectly. We define a 3D archive, with Topic, Interrogative Word and Question Length as features. The mutation operators for topics and interrogative words are analogous to those used in section 4 . For length, we simply prompt the Mutator to either “lengthen” or “shorten” the question. The preference model uses a Judge to compare answers from a Target (Llama 2-chat 7B) and a superior Oracle (Llama 2-chat 70B) to determine the fitness of questions based on the correctness of the responses. For more information, see section F.1 .

Results. In table 5 we compare Rainbow Teaming to a baseline that generates candidate questions from scratch rather than relying on existing questions in the archive. We observe that Rainbow Teaming achieves higher fitness, higher coverage (percentage of non-empty cells in the archive), and higher diversity in questions, indicating the importance of utilising previously discovered adversarial questions. Importantly, not relying on previous solutions leaves regions of the archive uncovered, particularly for short questions as seen in the example archives in appendix E . fig. 6 illustrates an example archive generated using Rainbow Teaming . Some example questions are also shown in section E.7 .

### 6.2 Cybersecurity

We apply Rainbow Teaming to cybersecurity, searching for adversarial prompts that elicit behaviour such as generating insecure code or providing assistance in orchestrating cyberattacks. We use a 2D archive with the 10 MITRE categories for cyberattack tactics ( MITRE, 2024 ) (e.g., “Exfiltration” or “Defense Evasion”) and prompt length divided into 10 equal bins. Our Mutator is an instruction-tuned Llama 2 70B model, mutating first for MITRE attack style, and then for prompt length. We use a binary Judge mechanism involving Llama 2-chat 70B and CodeLlama-34B Instruct models to evaluate generated prompts, as outlined in CyberSecEval ( Bhatt et al., 2023 ) . We provide further details in section F.2 .

Results. table 6 presents the results of a cybersecurity assessment for various target models on prompts generated by Rainbow Teaming . For all models, we successfully generate 10 × 10 10\times 10 archives that are fully identified as malicious, as classified by CyberSecEval ( Bhatt et al., 2023 ) . Human expert evaluation finds a lower ASR, with 0.94 0.94 and 0.92 0.92 for Llama 2-chat 7B and CodeLlama 7B Instruct, and 0.8 0.8 for both Llama 2-chat 70B and CodeLlama 34B Instruct. While Rainbow Teaming remains highly effective, the discrepancy between CyberSecEval and expert annotations suggests the need for a better cybersecurity-specific evaluation, which we hope will be the focus of future work.

## 7 Related Work