Rainbow Teaming: Open-Ended Generation of Diverse Adversarial Prompts

arxiv:2402.16822

Original source · Paper JSON

Retrieval, embeddings, benchmarks · 2024-02-26

Cites (3) · Cited by (7)

Complete cleaned Markdown · Source Markdown including bibliography

Read in parts

1. Part 1
2. Part 2
3. Part 3
4. Part 4

Contents

• R a i n b o w T e a m i n g : Open-Ended Generation of Diverse Adversarial Prompts
• Abstract
• 1 Introduction
• 2 Background
• 3 Rainbow Teaming
• 3.1 Prompt Features
• 3.2 Mutation Operator
• 3.3 Preference Model
• 4 Rainbow Teaming for Safety
• 4.1 Results
• 5 Enhancing Robustness with Synthetic Data
• 6 Rainbow Teaming for Other Applications
• 6.1 Question Answering
• 6.2 Cybersecurity
• 7 Related Work
• 8 Conclusion
• Acknowledgements
• Appendix A Limitations and Broader Impact
• Appendix B Algorithm Pseudocode
• B.1 MAP-Elites
• B.2 Rainbow Teaming Pseudocode
• Appendix C Extended Related Work
• C.1 Token-Level Attacks
• C.2 Adversarial Training
• Appendix D Adversarial Prompts as Stepping Stones
• Appendix E Additional Results
• E.1 Varying Model Sizes
• E.2 Role of System Prompts
• E.3 Human Evaluation
• E.4 Preference Model Ablation
• E.5 Full Evaluations
• E.6 Archive Visualisation
• E.7 Question Answering Examples
• Appendix F Additional Details for Preference Models
• F.1 Question Answering
• F.2 Cybersecurity
• Appendix G Inference Cost Analysis
• Appendix H Feature Descriptors
• H.1 Safety
• H.2 Question Answering
• H.3 Cybersecurity
• Appendix I Safety Experiment Prompts
• I.1 Llama Guard Evaluation Prompt
• I.2 Mutation Prompts
• I.3 System Prompt of Target LLM
• Appendix J Question Answering Experiment Prompts
• Appendix K Hyperparameters
• Instructions for reporting errors