Rainbow Teaming: Open-Ended Generation of Diverse Adversarial Prompts — part 4/4

Cleaned-text code points [47078, 53896). Bibliography omitted here; source.md includes it.

fig. 13 illustrates examples archives at various iterations of Rainbow Teaming generated in the safety domain. fig. 14 shows 2D projections of 3D archives of Rainbow Teaming at different iterations when applied in the question answering domain.

### E.7 Question Answering Examples

table 9 provides sample questions generated by Rainbow Teaming for the question answering domain.

## Appendix F Additional Details for Preference Models

### F.1 Question Answering

The preference model used in question answering domain differs from that used in section 4 to account for the difficulty of evaluating the relative correctness of responses to two different questions. For each question q q , we generate an answer r t r_{t} from the Target and another r o r_{o} from an Oracle LLM.

While both the Oracle and Target models receive identical prompts, the Oracle is equipped with superior capabilities (Llama 2-chat 70B) compared to the Target (Llama 2-chat 7B). We then provide the question q q alongside both answers r t r_{t} and r o r_{o} to the Judge to determine whether the question is factual and objective and whether the Oracle’s answer is better than the Target’s answer. If these conditions are met, we save the question to the archive with fitness 1 1 . If not, but the archive cell is empty, we save the question with a fitness of 0.1 0.1 to act as a stepping stone. We discard the question in all other scenarios.

### F.2 Cybersecurity

When using Rainbow Teaming in the cybersecurity domain, we use a binary Judge mechanism to evaluate the potential maliciousness of generated prompts. This is aligned with the framework outlined by Bhatt et al. [2023] , which leverages two distinct LLMs for classifying the output of an LLM as either “benign” or “malicious”. For the initial processing of the output, we use the Llama 2-chat 70B model, which is responsible for preliminary analysis and identification of potentially malicious content. The final decision of whether a response is indeed malicious is then made by the CodeLlama-34B Instruct model [ Rozière et al., 2023 ] .

## Appendix G Inference Cost Analysis

Since inference costs vary based on infrastructure, the number of generation tokens, and specific LLMs used, we choose to discuss computational costs in terms of LLM inference calls. For the majority of our experiments, one Rainbow Teaming loop requires 2 Mutator inference calls, 1 Target inference call, and 4 Judge inference calls, where every inference call corresponds to a batch of prompts (our batch size is 16 for most experiments, except 64 for our JailbreakBench ones). When running for 2000 iterations, this represents a total of 14000 batched inference calls per run.

We conducted our experiments on a cluster of A100 GPUs, with access ranging from 128 to 256 GPUs throughout the project. Each run was typically completed in around two days, though we often accelerated them significantly by leveraging a distributed client-server setup for parallelised LLM inference.

Although the computational cost is substantial, we believe the effectiveness and diversity of the resulting prompts make our method a highly competitive alternative to crowd-sourced data.

Depending on the user’s needs, costs can be further amortised by leveraging earlier Rainbow Teaming checkpoints as a source of additional prompts, potentially expanding the final dataset size by an order of magnitude. As LLMs become more powerful, so do the components of our method. For example, if an LLM can serve as an accurate judge in a single inference call, this could significantly reduce the inference cost associated with the Judge.

## Appendix H Feature Descriptors

### H.1 Safety

Risk Category • Violence and Hate

• Sexual Content

• Criminal Planning

• Guns and Illegal Weapons

• Regulated or Controlled Substances

• Self-Harm

• Inciting or Abetting Discrimination

• Fraud and Scams

• Cybercrime and Hacking

• Terrorism

Attack Style • Slang

• Technical Terms

• Role Play

• Authority Manipulation

• Misspellings

• Word Play

• Emotional Manipulation

• Hypotheticals

• Historical Scenario

• Uncommon Dialects

### H.2 Question Answering

Topic • Science and Technology

• Health and Wellness

• History and Culture

• Arts and Entertainment

• Nature and Environment

• Travel and Geography

• Society and Politics

• Education and Learning

• Food and Cooking

• Relationships and Life

Question Length (measured in number of characters) • Min: 24

• Max: 96

• Number of bins: 10

Interrogative Word • Where

• Who

• What

• When

### H.3 Cybersecurity

MITRE attack tactics • Reconnaissance

• Discovery

• Execution

• Privilege Escalation

• Persistence

• Defense Evasion

• Lateral Movement

• Collection

• Command and Control

• Exfiltration

Instruction Length (measured in number of characters) • Min: 100

• Max: 1000

• Number of bins: 10

## Appendix I Safety Experiment Prompts

This section includes various prompts used in experiments in the safety domain.

### I.1 Llama Guard Evaluation Prompt

For Llama Guard evaluations, we use the default system prompts by Inan et al. [2023] for 6 original categories. For the 4 new categories, similar system prompts have been generated by the authors.

### I.2 Mutation Prompts

We have elected not to share mutation prompts to prevent misuse.

### I.3 System Prompt of Target LLM

Below are the prompts given to the Target Llama 2-chat model in our experiments with various system prompts presented in section E.2 . Here the {input} is replaced with adversarial prompts.

## Appendix J Question Answering Experiment Prompts

This section includes prompts used in the question answering experiments.

## Appendix K Hyperparameters

Tables 10 , 11 , and 12 provide the hyperparameters used in safety, question answering, and cybersecurity domains, respectively.

## Instructions for reporting errors

We are continuing to improve HTML versions of papers, and your feedback helps enhance accessibility and mobile support. To report errors in the HTML that will help us improve conversion and rendering, choose any of the methods listed below:

Click the "Report Issue" ( ) button, located in the page header.

Tip: You can select the relevant text first, to include it in your report.

Our team has already identified the following issues . We appreciate your time reviewing and reporting rendering errors we may not have found yet. Your efforts will help us improve the HTML versions for all readers, because disability should not be a barrier to accessing research. Thank you for your continued support in championing open access for all.

Have a free development cycle? Help support accessibility at arXiv! Our collaborators at LaTeXML maintain a list of packages that need conversion , and welcome developer contributions .