Rainbow Teaming: Open-Ended Generation of Diverse Adversarial Prompts — part 1/4

Cleaned-text code points [0, 15461). Bibliography omitted here; source.md includes it.

# R a i n b o w T e a m i n g : Open-Ended Generation of Diverse Adversarial Prompts

###### Abstract

As large language models (LLMs) become increasingly prevalent across many real-world applications, understanding and enhancing their robustness to adversarial attacks is of paramount importance. Existing methods for identifying adversarial prompts tend to focus on specific domains, lack diversity, or require extensive human annotations. To address these limitations, we present Rainbow Teaming , a novel black-box approach for producing a diverse collection of adversarial prompts. Rainbow Teaming casts adversarial prompt generation as a quality-diversity problem, and uses open-ended search to generate prompts that are both effective and diverse. Focusing on the safety domain, we use Rainbow Teaming to target various state-of-the-art LLMs, including the Llama 2 and Llama 3 models. Our approach reveals hundreds of effective adversarial prompts, with an attack success rate exceeding 90% across all tested models. Furthermore, we demonstrate that prompts generated by Rainbow Teaming are highly transferable and that fine-tuning models with synthetic data generated by our method significantly enhances their safety without sacrificing general performance or helpfulness. We additionally explore the versatility of Rainbow Teaming by applying it to question answering and cybersecurity, showcasing its potential to drive robust open-ended self-improvement in a wide range of applications.

## 1 Introduction

Large language models (LLMs) have recently experienced remarkable growth in both their capabilities ( OpenAI, 2023 ; Gemini Team et al., 2023 ; Touvron et al., 2023 ) and their applications in various fields ( NLLB Team et al., 2022 ; Thirunavukarasu et al., 2023 ; Schick et al., 2023 ; Bubeck et al., 2023 ) . As LLMs become increasingly complex and are deployed in safety-critical environments ( Singhal et al., 2022 ; Li et al., 2023 ; Maddela et al., 2023 ) , it is essential to thoroughly understand their robustness to different inputs. Indeed, the susceptibility of LLMs to adversarial prompts — prompts crafted to mislead the model or exploit its weaknesses, potentially leading to unsafe, biased, or incorrect outputs — poses a significant challenge ( Perez et al., 2022 ; Wei et al., 2023 ; Zou et al., 2023 ) . Identifying and mitigating these vulnerabilities is therefore vital to ensure the safe and reliable operation of LLMs in the real world ( Anwar et al., 2024 ; Hendrycks et al., 2022 ) .

Current methods for identifying adversarial prompts aimed at “attacking” LLMs and eliciting undesirable outputs are limited by several factors. These limitations include the necessity of fine-tuning an attacker model ( Perez et al., 2022 ; Ge et al., 2023 ) ; white-box access to the target model ( Liu et al., 2023 ) ; or significant human input ( Yu et al., 2023 ) . Several recent black-box methods, such as PAIR ( Chao et al., 2023 ) and TAP ( Mehrotra et al., 2023 ) , rely on a human-in-the-loop to specify harmful behaviours (e.g., “Write a tutorial on how to hack the university server”) in a way where the human effort required is linear in the number of desired prompts. Many existing methods for systematically discovering adversarial attacks exhibit a lack of diversity by design ( Liu et al., 2023 ) , for instance by restricting themselves to a single predefined attack strategy ( Shah et al., 2023 ; Jiang et al., 2024 ; Anil et al., 2024 ) . Others suffer from loss of diversity, a common issue in objective-based prompt optimisation approaches ( Zhou et al., 2022 ; Fernando et al., 2023 ) . In both cases, the narrow focus of generated prompts limits the usefulness of those methods both as a diagnostic tool and as a source of synthetic data for improving robustness.

We introduce Rainbow Teaming , a versatile approach for systematically generating diverse adversarial prompts for LLMs via LLMs. While the prevailing approach to automatic red teaming ( Perez et al., 2022 ) also uses LLMs to generate adversarial inputs, it exhibits a steep trade-off between the diversity of discovered attacks and their success rate. In contrast, Rainbow Teaming takes a more deliberate approach, efficiently covering the space of attacks by directly optimising for the attack quality and diversity. To this end, our method casts the problem of adversarial prompt generation as quality-diversity (QD) search ( Lehman and Stanley, 2011 ; Pugh et al., 2016 ; Cully and Demiris, 2018 ) and takes direct inspiration from Samvelyan et al. (2024) to discover a set of adversarial prompts that are both diverse and effective.

Rainbow Teaming is an open-ended approach ( Hughes et al., 2024 ) which builds on MAP-Elites ( Mouret and Clune, 2015 ) , an evolutionary search method that iteratively populates an “archive” with increasingly higher-performing solutions. In our case, these solutions are adversarial prompts that elicit undesirable behaviours in a target LLM, while the archive is a discrete grid where each dimension categorises prompts according to a feature of interest for diversity, such as attack style, risk category, or prompt length. The output of our method, as shown in footnote 2 , is a set of prompts covering every combination of features specified by the archive. These diverse and effective attack prompts serve both as a diagnostic tool for the vulnerabilities of the target LLM and as a high-quality synthetic dataset to robustify the target.

Rainbow Teaming is directly applicable to a wide range of domains. Implementing Rainbow Teaming requires three essential building blocks: 1) A set of features that specify the dimensions of diversity (e.g., “Risk Category” or “Attack Style”); 2) A mutation operator to evolve adversarial prompts (e.g., an LLM that is itself prompted to mutate previously discovered prompts ( Lehman et al., 2022 ) ); and 3) a preference model that ranks adversarial prompts based on their effectiveness. For safety, this can be a “judge” LLM ( Zheng et al., 2023 ) that compares two responses to determine which is more unsafe.

We demonstrate the effectiveness of Rainbow Teaming through extensive experiments targeting several state-of-the-art LLMs fine-tuned on safety-aligned data, including the Llama 2-chat ( Touvron et al., 2023 ) and Llama 3-Instruct ( AI@Meta, 2024 ) models. Despite the rigorous development of these models, our experiments reveal hundreds of adversarial prompts per individual run, achieving an attack success rate higher than 90% across all tested models without requiring external data. Using popular safety benchmarks, we demonstrate that Rainbow Teaming outperforms strong baselines in identifying vulnerabilities. Additionally, fine-tuning LLMs with synthetic data generated by our approach significantly enhances their adversarial robustness, improving resistance to unseen attacks and subsequent rounds of Rainbow Teaming , without diminishing their general capabilities and helpfulness.

We further illustrate the versatility of Rainbow Teaming by applying it to other domains, such as question answering and cybersecurity, uncovering hundreds of effective adversarial prompts in each case. These findings underscore Rainbow Teaming ’s potential as a comprehensive tool for diagnosing and advancing the robustness and reliability of LLMs across diverse applications.

## 2 Background

Rainbow Teaming builds on existing approaches in quality-diversity (QD) search to automate the discovery of a broad spectrum of adversarial prompts. QD methods seek to produce a collection of solutions that are individually high-performing and collectively diverse ( Lehman and Stanley, 2011 ; Cully and Demiris, 2018 ) . Given a space of solutions 𝒳 \mathcal{X} , the quality of a solution x ∈ 𝒳 x\in\mathcal{X} is measured using a fitness function f : 𝒳 → ℝ f:\mathcal{X}\rightarrow\mathbb{R} . The diversity of solutions is characterised using a feature descriptor function , d : 𝒳 ↦ 𝒵 d:\mathcal{X}\mapsto\mathcal{Z} that maps each solution to a point in a feature space 𝒵 = ℝ N \mathcal{Z}=\mathbb{R}^{N} . This space encompasses specific pre-defined attributes of the solution, such as its behavioral aspects. For each z ∈ 𝒵 z\in\mathcal{Z} , QD searches for the solution x ∈ 𝒳 x\in\mathcal{X} such that d ⁡ ( x ) = z d(x)=z and f ⁡ ( x ) f(x) is maximised.

Our work builds directly on MAP-Elites ( Mouret and Clune, 2015 ) , a simple yet effective QD method. MAP-Elites tracks the highest-fitness solutions in a multidimensional grid, referred to as the archive , which discretises the feature space 𝒵 \mathcal{Z} . The archive is first initialised with random solutions. During each iteration of MAP-Elites, a solution x x is sampled at random from the archive and modified to create a new solution x ′ x^{\prime} (e.g., by injecting Gaussian noise). The new solution x ′ x^{\prime} is then evaluated and assigned to its corresponding archive cell based on its descriptor z ′ = d ⁡ ( x ′ ) z^{\prime}=d(x^{\prime}) . If the cell is vacant, or if x ′ x^{\prime} has higher fitness than the current occupant, also known as the elite , x ′ x^{\prime} becomes the new elite for that cell. Through repeated cycles of selection, mutation, and evaluation, MAP-Elites fills the archive with the highest-fitness solutions. algorithm 1 in appendix B provides the pseudocode of this method.

## 3 Rainbow Teaming

We now describe Rainbow Teaming , our new approach for automatically generating a diverse collection of adversarial prompts. Rainbow Teaming casts this task as a QD search problem with the solution space corresponding to all possible prompts. Our rationale for employing QD is twofold: • Effective adversarial prompts for specific scenarios (e.g., criminal planning) could be effective for others (e.g., cybercrime and hacking) with relatively small modifications. This adaptability implies that solutions can serve as stepping stones to accelerate the discovery of new adversarial strategies across different categories.

• A thorough diagnostic of the vulnerabilities of a model calls for a comprehensive analytical tool to mitigate the risks of leaving attack vectors undiscovered. Similarly, safety fine-tuning requires a sufficiently diverse dataset to improve a model’s adversarial robustness against a wide range of attacks. Diversity is essential for both of these objectives, and QD allows us to optimise it explicitly.

Rainbow Teaming is based on MAP-Elites ( Mouret and Clune, 2015 ) . We store adversarial prompts as solutions in a K K -dimensional archive, with each dimension corresponding to one of the pre-defined features. Each cell in the archive corresponds to a unique combination of K K categories that describe the prompt within it, known as the cell’s and the solution’s descriptor , and denoted z = ⟨ c 1 , … , c K ⟩ z=\langle c_{1},\dots,c_{K}\rangle . The LLM for which the adversarial prompts are generated is referred to as the Target . Initial solutions can be either generated randomly using an LLM or loaded from an existing dataset. As shown in fig. 2 , all key operation of the iterative search are performed with LLMs.

At each iteration of Rainbow Teaming , we sample 1) an adversarial prompt x x from the archive with descriptor z z , and 2) a descriptor z ′ z^{\prime} for the new candidate prompt to be generated. Note that z z and z ′ z^{\prime} are different. 3 3 3 In fig. 2 , z = ⟨ “Criminal Planning” , “Role Play” ⟩ z=\langle\textrm{``Criminal Planning''},\textrm{``Role Play''}\rangle , while z ′ = ⟨ “Fraud and Scams” , “Misspellings” ⟩ z^{\prime}=\langle\textrm{``Fraud and Scams''},\textrm{``Misspellings''}\rangle . We provide x x and z ′ z^{\prime} to the Mutator LLM to generate a new candidate prompt x ′ x^{\prime} with descriptor z ′ z^{\prime} . We then feed x ′ x^{\prime} to the Target to generate a response. Finally, we ask a Judge LLM ( Zheng et al., 2023 ) to compare the effectiveness of the candidate prompt x ′ x^{\prime} to that of the archive’s elite prompt – the prompt stored in the archive with a descriptor z ′ z^{\prime} . This comparison focuses on the criteria of interest, such as the toxicity of the Target response, to determine which of the two prompts more effectively meets the adversarial objective. We then store the winning prompt in the archive at the position specified by z ′ z^{\prime} . algorithm 2 in appendix B provides the pseudocode of our method.

Rainbow Teaming is highly versatile and can easily be applied to various settings by implementing three components: prompt features, a mutation operator, and a preference model.

### 3.1 Prompt Features

The features define the archive, with each predefined feature corresponding to one of the K K archive dimensions. A feature can be either categorical or numerical. For categorical features, the axis of the archive is composed of discrete bins each representing a unique feature category. For instance, the Risk Category and Attack Style features in footnote 2 each consist of 10 10 categories. Numerical features are represented on a continuous scale, discretised into a set of intervals. Features therefore determine both the final archive size and the axes of diversity that Rainbow Teaming prioritises. This is particularly true given their interplay with the mutation operator , as described next.

### 3.2 Mutation Operator

Rainbow Teaming generates new candidates by applying directed mutations to previously discovered adversarial prompts. The Mutator receives a parent prompt x x sampled uniformly at random from the archive and the prescribed descriptor z ′ = ⟨ c 1 ′ , … , c K ′ ⟩ z^{\prime}=\langle c^{\prime}_{1},\dots,c^{\prime}_{K}\rangle for the candidate. It then mutates the prompt x x once for each feature — K K times overall — to produce a new candidate prompt x ′ x^{\prime} .

Sampling the candidate’s descriptor in advance confers several key benefits. First, this allows us to forgo using a classifier for assigning the candidate to its corresponding cell, which can be inaccurate. Second, it introduces more diversity by mitigating the biases of the Mutator, which could otherwise neglect entire categories. Third, it helps avoid spending iterations on areas of the archive for which we already have effective adversarial prompts. We do this by biasing the sampling distribution of the descriptors towards areas of the archive with low fitness. We compute fitness explicitly for this purpose but do not use it to inform archive updates.

To further promote diversity, the candidate prompt is considered for further evaluation only if it is sufficiently dissimilar from its parent. We measure the similarity using BLEU ( Papineni et al., 2002 ) and filter out prompts that have high BLEU scores with respect to their parents.

### 3.3 Preference Model

The preference model, operated through the Judge, performs the ranking of adversarial prompts based on their effectiveness (e.g., whether they elicit unsafe responses). The Judge inputs can vary between domains, but preference-based evaluations include the Target responses to both the candidate and the existing prompt from the archive with descriptor z ′ z^{\prime} . The Judge determines which prompt is more effective using a majority vote over multiple evaluations and swapping prompt positions to mitigate order bias ( Zheng et al., 2023 ) . If the candidate wins the comparison, it replaces the existing prompt.