Rainbow Teaming: Open-Ended Generation of Diverse Adversarial Prompts — part 3/4

Cleaned-text code points [31088, 47078). Bibliography omitted here; source.md includes it.

Adversarial Attacks on LLMs. Rainbow Teaming relates most closely to prompt-level attacks which rely on strategies such as misspellings, prompting in foreign languages ( Yong et al., 2023 ) , or persona-modulation ( Shah et al., 2023 ) to jailbreak LLMs. Perez et al. (2022) use an LLM and a brute-force approach to automatically discover prompt-level attacks, but this approach can suffer from mode collapse and does not always generate a diverse set of prompts. Meanwhile, Liu et al. (2023) propose a white-box method that refines hand-crafted attack prompts using a mix of genetic algorithms and LLM-based mutations. However, they focus on optimising a single solution rather than a diverse population. The closest works to our own are PAIR ( Chao et al., 2023 ) and Tree of Attacks with Pruning (TAP) ( Mehrotra et al., 2023 ) — two black-box methods for automatically discovering prompt-level attacks by using an LLM to iteratively generate candidates. However, both methods are designed to jailbreak the model with respect to a single task rather than across a range of diverse risk categories and attack styles. In contrast, our work uses quality-diversity search to automatically discover attacks covering a diverse set of risks and attack strategies. Although evolutionary algorithms have previously been used for adversarial attacks on LLMs ( Liu et al., 2023 ; Lapid et al., 2023a ; Chao et al., 2023 ) , this work is the first to apply a quality-diversity framework ( Lehman and Stanley, 2011 ; Cully and Demiris, 2018 ) in this area. Unlike most evolutionary algorithms (e.g., genetic algorithms), which evolve a single optimal solution, quality-diversity approaches generate a wide variety of distinct, high-quality solutions.

Open-Endedness and LLMs. Rainbow Teaming builds on the ability of LLMs to act as a powerful mutation operator over language inputs, one that adheres to the underlying structure of natural language ( Lehman et al., 2022 ) . Several recent methods exploit this capability of LLMs in order to perform an efficient novelty-driven evolutionary search in the language space, leading to the discovery of potentially open-ended repertoires of solutions ( Chen et al., 2023 ; Fernando et al., 2023 ; Meyerson et al., 2023 ) . Closest to our approach is QDAIF ( Bradley et al., 2023 ) which similarly uses LLMs for QD search in order to generate a diverse archive of LLM outputs. Rainbow Teaming is different from QDAIF in several important factors. First, we search for and archive diverse prompts for the target LLMs, whereas QDAIF archives diverse responses from it — a separate problem altogether. While QDAIF focuses purely on generating diverse outputs for creative writing, our method seeks to find a diverse set of adversarial prompts. QDAIF relies on a score-based fitness function (log probability of the token generation), whereas Rainbow Teaming uses a preference-based judge for performing updates to the archive. Rainbow Teaming additionally incorporates parent-child similarity filtering to preserve the linguistic diversity of the prompts.

An extended related work section is provided in appendix C .

## 8 Conclusion

In this work, we introduce Rainbow Teaming , a novel approach for the automatic generation of diverse adversarial prompts for LLMs. By leveraging quality-diversity search, Rainbow Teaming efficiently explores the space of potential adversarial attacks, resulting in a diverse archive of prompts that highlight the vulnerabilities of LLMs. Our extensive experiments with multiple models, such as Llama 3-Instruct and Llama 2-chat, and across various domains, including safety, question answering, and cybersecurity, demonstrate the generality of Rainbow Teaming . Moreover, the synthetic data generated through Rainbow Teaming can be utilised for fine-tuning LLMs, thereby enhancing their resilience against further adversarial attacks without compromising their general performance. This illustrates the potential of Rainbow Teaming as a means for the continuous, open-ended self-improvement of LLMs, with minimal human intervention. Future work with Rainbow Teaming involves extending its application beyond LLMs to areas such as vision and multi-modal AI systems. Moreover, incorporating Rainbow Teaming into the fine-tuning stages of LLM development presents an opportunity to consistently strengthen their defences against adversarial attacks.

We discuss the limitations and broader impact of our work in appendix A .

## Acknowledgements

We extend our gratitude to Alex Havrilla, Robert Kirk, Maya Pavlova, Suyu Ge, Joshua Saxe, and Aaron Grattafiori for their insightful discussions and feedback on our work. We also thank Sten Sootla, Lovish Madaan, Anthony Hartshorn, Jeremy Reizenstein, and Henry Estela, for their assistance in conducting experiments. We extend our deepest gratitude to Nicola Cancedda and Naila Murray for their invaluable support and guidance, which were crucial to this work.

Andrei was partially funded by a Fonds de recherche du Québec doctoral training scholarship.

## Appendix A Limitations and Broader Impact

Despite many advantages of Rainbow Teaming , its current implementation has several limitations. First, the features that define the archive and its categories are pre-defined and fixed. In future work, it would be interesting to extend our approach to discover features and categories automatically. Another limitation of Rainbow Teaming is that the number of prompts it can generate is constrained by the grid size. While this is due to using MAP-Elites as the base QD algorithm, we note that even the current setting allows generating hundreds of adversarial prompts from a single run and this can be extended by providing additional features or categories or storing several values within the same archive cell.

Unlike simpler adversarial attack methods [ Chao et al., 2023 ] , Rainbow Teaming requires extensive computational resources. Furthermore, its undirected, open-ended approach is less likely to produce a prompt for a specific behaviour (e.g., writing a fake news article about a specific public figure). While these attributes can be considered limitations, we highlight that because of them, Rainbow Teaming is less likely to be used for malicious purposes. The primary value of Rainbow Teaming lies in its potential to identify and address robustness issues in LLMs, contributing to their responsible development and deployment.

Ultimately, we believe Rainbow Teaming to be a powerful tool in improving the robustness of LLMs to adversarial attacks and see the prompts it generates as a valuable complement to crowd-sourced data.

## Appendix B Algorithm Pseudocode

### B.1 MAP-Elites

algorithm 1 provides a pseudocode of MAP-Elites method [ Mouret and Clune, 2015 ] described in section 2 .

### B.2 Rainbow Teaming Pseudocode

algorithm 2 provides a pseudocode of Rainbow Teaming described in section 3 .

Throughout this work, we use BLEU score [ Papineni et al., 2002 ] as the similarity metric s ​ i ​ m sim . In the safety domain, we use the probability of Llama Guard categorising a response as unsafe as the fitness function f f . The fitness function is used for biasing the sampling of descriptor d d but not for updating the archive.

For clarity, the algorithm shows the Rainbow Teaming loop over a single prompt x x , but the process can be batched to reduce wall clock time. In practice, we use batch sizes between 16 and 64.

## Appendix C Extended Related Work

### C.1 Token-Level Attacks

Token-level attacks circumvent the LLM’s defences against generating undesirable responses by adding adversarial tokens to a malicious prompt. Such methods originally required white-box access to the LLM [ Zou et al., 2023 ] , but that assumption has since been relaxed using black-box optimisation [ Lapid et al., 2023b , Maus et al., 2023 ] . Token-level attacks have proven effective, but brittle to perturbations [ Robey et al., 2023 ] . Although Rainbow Teaming could be adapted to create token-level attacks by integrating the appropriate attack categories and prompts, we restrict this study to prompt-level attacks given that prompt-level attacks are more interpretable and harder to detect.

### C.2 Adversarial Training

Rainbow Teaming ’s approach parallels other forms of adversarial training, which prioritises training on tasks or data points where the model performs poorly. In reinforcement learning (RL), methods such as active domain randomisation [ Mehta et al., 2020 , Raparthy et al., 2020 ] and regret-based unsupervised environment design [ Dennis et al., 2020 , Jiang et al., 2021 , Parker-Holder et al., 2022 , Samvelyan et al., 2023 ] search for training tasks where the agent performs poorly in terms of absolute task performance or regret, respectively. Regret-based prioritisation has been shown to hold robustness guarantees at convergence and carry the benefit of avoiding unsolvable tasks (which always result in zero regret). The fitness score used by Rainbow Teaming coincides with regret [ Savage, 1951 ] , as a high fitness here implies the existence of another prompt that elicits a less undesirable response, as evaluated by the Judge. Similarly, many active learning and automatic curriculum learning methods in supervised learning focus training on examples maximising error metrics derived from the model’s predictions [ Graves et al., 2017 , Mindermann et al., 2022 , Evans et al., 2023 ] . Dynabench [ Kiela et al., 2021 ] extends this paradigm by querying humans-in-the-loop for adversarial examples. Many methods in scenario generation also closely relate to Rainbow Teaming , including recent approaches using QD search to find adversarial environments that induce poor behaviour in fully-automated or mixed-autonomy systems [ Fontaine et al., 2021 , Fontaine and Nikolaidis, 2022 , Bhatt et al., 2022 ] . This extends to recent work applying QD to multi-agent RL [ Samvelyan et al., 2024 ] , which inspired our method.

## Appendix D Adversarial Prompts as Stepping Stones

fig. 7 provides a qualitative example of how the directed mutation in Rainbow Teaming can produce diverse adversarial prompts from a single common ancestor.

## Appendix E Additional Results

### E.1 Varying Model Sizes

fig. 8 presents the ASR of Rainbow Teaming when applied to Llama 2-chat models with 7B, 13B, and 70B parameters across 2000 iterations, using GPT-4 and Llama Guard for evaluation. Archives generated through Rainbow Teaming demonstrate 90% or higher ASR across all model sizes, as measured using GPT-4 and Llama Guard evaluators.

### E.2 Role of System Prompts

While our main experiments provide the prompts to the Target as is (within appropriate instruction tokens), we additionally analyse incorporating two system prompts . The legacy system prompt is designed to emphasise both safety and helpfulness . 5 5 5 It was initially released with Llama 2 but has since been deprecated due to its high false refusal rate. See the change here . The helpful system prompt is a handcrafted variant of the legacy prompt that focuses on helpfulness without explicitly emphasising safety. All system prompts are provided in section I.3 .

The effectiveness of Rainbow Teaming when using these different system prompts is presented in table 7 . Our results indicate the inclusion of a system prompt emphasising safety diminishes the success rate of adversarial attacks to 51% / 74%, according to GPT-4 and Llama Guard evaluations, respectively. However, using this system prompt makes the model overly conservative, occasionally refusing to answer benign questions that appear unsafe. On the other hand, the helpful system prompt, remains vulnerable to attacks, with 82% / 93% ASR, yet still offers improved robustness compared to not using a system prompt at all, which sees 92% / 95% ASR. The Llama 2-chat 7B model has been safety-aligned regardless of the system prompt, but its robustness is highly dependent on this variable.

### E.3 Human Evaluation

table 8 shows the ASR of 100 prompts, as evaluated by GPT-4, Llama Guard and human annotators. The inter-evaluator agreement is also provided.

The 100 prompts were randomly sampled from a combination of archives targeting Llama 2-chat 7B or 70B, using either no system prompt or the Legacy prompt. This was done to ensure sufficient prompt variety. The annotations were done within a single session by four of the authors following the same harm taxonomy as Llama Guard. The annotators were not informed of the data mix or the expected ASR as evaluated by either Llama Guard or GPT-4 at the time of annotation to mitigate bias. All annotations were performed through an interactive spreadsheet, where authors tick a checkbox to label the model response as “GOOD BOT” (when the target model refused to answer or answered in a safe manner) or “BAD BOT” (when the model answered in a way that is violating according to the harm taxonomy).

### E.4 Preference Model Ablation

We investigate the role of the preference model used for Rainbow Teaming . One option is to prompt an LLM to act as a pairwise comparison Judge [ Zheng et al., 2023 ] . The Judge takes as input the response to the candidate prompt and the response to the prompt already in the archive, and is instructed to say which of the two is more harmful. As recommended by Zheng et al. [2023] , we use position swapping (evaluating twice for each order of the prompt-response pairs) to reduce positional bias, and few-shot prompting to improve consistency. We refer to this variant as “Comparison-based (Judge)” and use it as default throughout the paper.

We compare the default version of Rainbow Teaming with a variant that uses the Llama Guard probability of classifying the response as “unsafe” as a preference model. In this case, we perform an archive substitution if the probability for the candidate response is higher than that of the existing response. We refer to this Rainbow Teaming variant as “Score-based (No Judge)”.

In our evaluation, as shown in Figure 9 , the score-based baseline achieves a higher Llama Guard-evaluated ASR, aligning with its optimisation objective. However, it falls short in GPT-4-evaluated ASR, suggesting overfitting to Llama Guard scores, indicative of reward hacking [ Skalse et al., 2022 ] . Qualitatively, we find that the adversarial prompts produced by the score-based method are also of lower quality. We also show the number of archive updates for the two variations of Rainbow Teaming . We observe that the No Judge baseline quickly maximising the Llama Guard score (capped to 1.0 1.0 ) leads to sparse updates thereafter. In contrast, the Judge-based variant continues to refine the quality of the adversarial prompts in the archive, indicated by ongoing archive updates, even after filling the archive with successful prompts. This underscores the advantage of Rainbow Teaming ’s open-ended search process over a purely score-driven approach.

Note that the performance differences between Rainbow Teaming results here and in other parts of the manuscript arise from variations in the experimental setup. In this specific experiment, we use Anthropic Harmless as the seed dataset with slightly different mutation prompts, and two risk category names have been updated.

### E.5 Full Evaluations

fig. 10 presents the ASR of Rainbow Teaming when applied to Llama 2-chat 7B [ Touvron et al., 2023 ] , Llama 3-Instruct 8B [ AI@Meta, 2024 ] , Mistral 7B [ Jiang et al., 2023 ] and Vicuna 7B v1.5 [ Chiang et al., 2023 ] models across 2000 iterations, using both GPT-4 and Llama Guard for evaluation. fig. 11 shows the performance of Rainbow Teaming against No Stepping Stones and Same Cell Mutations baselines, using GPT-4 and Llama Guard for evaluations. In fig. 12 we report the performance of our approach targeting Llama 2-chat 7B model before and after performing SFT on Rainbow Teaming -generated data.

### E.6 Archive Visualisation