Promptbreeder: Self-Referential Self-Improvement Via Prompt Evolution — part 4/4

Cleaned-text code points [47951, 51843). Bibliography omitted here; source.md includes it.

#### K.1.18 Translation English-German

#### K.1.19 Translation English-Spanish

#### K.1.20 Translation English-French

⬇

#### K.1.21 Sentiment Analysis

#### K.1.22 Sentence Similarity

#### K.1.23 Word in Context

## Appendix L Ablations

We performed ablation to measure the impact of various self-referential components of Promptbreeder. We investigated the following mutation operators and mechanisms: • Random initial prompts

The original problem specification for the dataset is used instead of generating an initial task-prompt using the mutation prompt + thinking style + problem specification.

• Random initial mutation prompts The mutation-prompt ”Please summarize and improve the following instruction:” is used instead of randomly selecting a mutation-prompt from the list.

• Prompts from context (Lamarckian) The Lamarckian mutation operator that generates a task-prompt from a correct context is replaced with the default zero-/first-order prompt mutation operation (50:50 chance of one or the other)

• Meta-mutation (mutating mutation-prompts) When meta-mutation would normally take place the default zero-/first-order prompt mutation operation is performed (50:50 chance of one or the other)

For each dataset and each ablation, we use a population of 10 for 200 evaluations (equivalent to 20 generations, similar to larger experiments in this paper) and compare to the complete algorithm with the same population size and no ablations. To measure how effective an ablated operation is, we determine the proportion of evaluations in the ablation that were higher than the baseline evaluations at each generation, and sum these over all generations in the run. The results in Figure 4 show that in most cases all the mutation operators have a positive impact on fitness, with the Random Initial Prompts having the largest positive impact across all datasets.

We also investigated the influence of different mutation operators on the ETHOS hate speech detection dataset ( Mollas et al., 2022 ) with the under-specified problem specification "Solve the Problem" (in contrast to the standard problem specification "Determine whether a text contains hate speech" ). Promptbreeder achieved a score of 81.6 % 81.6\% . The greatest deterioration happens when removing the Lamarckian ‘from context to prompt’ mutation method which induces the instruction from an example of the correct working out ( 64.6 % 64.6\% ). The second greatest detriment to performance happens when removing random initialization of mutation prompts, random initialization of prompts, and hyper-mutation of mutation prompts simultaneously, leaving only context mutation ( 68.7 % 68.7\% ). Adding back online mutation increases performance back to 70.4 % 70.4\% and adding random mutation prompts brings this back up to 73.7 % 73.7\% . This demonstrates the interplay and importance of Promptbreeder’s diverse set of mutation operators.

## Instructions for reporting errors

We are continuing to improve HTML versions of papers, and your feedback helps enhance accessibility and mobile support. To report errors in the HTML that will help us improve conversion and rendering, choose any of the methods listed below:

Click the "Report Issue" ( ) button, located in the page header.

Tip: You can select the relevant text first, to include it in your report.

Our team has already identified the following issues . We appreciate your time reviewing and reporting rendering errors we may not have found yet. Your efforts will help us improve the HTML versions for all readers, because disability should not be a barrier to accessing research. Thank you for your continued support in championing open access for all.

Have a free development cycle? Help support accessibility at arXiv! Our collaborators at LaTeXML maintain a list of packages that need conversion , and welcome developer contributions .