Promptbreeder: Self-Referential Self-Improvement Via Prompt Evolution — part 3/4
Cleaned-text code points [31962, 47951). Bibliography omitted here; source.md includes it.
We introduced Promptbreeder (PB), a self-referential self-improving system that can automatically evolve effective domain-specific prompts for a domain at hand. PB is self-referential in that it not only evolves task-prompts, but it also evolves mutation-prompts that govern the way PB modifies task-prompts. Thus, it is not only improving prompts but it also improves the way it is improving prompts. Going forward, it could be interesting to use the LLM itself to assess and promote the diversity of generated prompts ( Zhang et al., 2023a , see) , or to use it to determine the fitness of a whole ‘‘thought process’’, e.g. an N-prompt strategy where prompts are conditionally applied rather than unconditionally applied as in Promptbreeder. For example, a more complex ‘‘thought process’’ is to use PB in self-play mode to evolve pre-prompts for LLM-based policies that compete with each other, i.e., in a competitive Socratic 5 5 5 https://princeton-nlp.github.io/SocraticAI/ dialog. PB remains limited compared to the open-endedness of human thought processes. First, the topology of prompting remains fixed (see Figure 2 )—we only adapt the prompt content not the prompting algorithm itself. One interpretation of thought is that it is a reconfigurable open-ended self-prompting process. If so, how does one develop complex thought strategies? Clearly it is necessary to generate and evaluate them, and whilst a simple evolutionary process provides one framework in which a thought strategy could be evolved, our actual human experience suggests multiple overlapping hierarchical selective processes at play. Moreover, in addition to language, human thought involves intonation, imagery, etc., in a multimodal system. We believe PB points to an exciting future where increasingly open-ended self-referential self-improvement systems can directly use language as the substrate for improvement instead of relying on any parameter updates. This is intriguing, as this approach will likely continue to scale with ever larger and more capable LLMs in the future. #### Acknowledgments We thank Edward Hughes and Tom Schaul for feedback on an early draft of the paper. We also thank Tom Schaul, Chengrun Yang, and Denny Zhou for fruitful discussions, as well as Gavin Buttimore, Simon Green, Keith Anderson, Joss Moore, Ollie Purkiss, John Quan, and Francesco Visin for their support in running some of the experiments. ## Appendix A Glossary Estimation of Distribution Algorithm An optimization algorithm that iteratively refines a probabilistic model of promising solutions, often using the whole population as a guide. Also knows as Roulette-Wheel Selection, an individual is chosen in proportion to its fitness in the population. The text prompt which when concatenated to the task-prompt is intended to produce a continuation which is an improved task-prompt. The initial text description of the problem which could be used as the initial task-prompt. The user can make their best attempt to produce an effective problem description, which is the starting point of Promptbreeder. A set of task-prompts and rules for their application at inference time during a fitness evaluation. In the minimal case the prompt strategy is just a single task-prompt. Typically our prompt strategies consisted of two sequentially applied task-prompts. Used interchangeably to mean the output of the LLM on a specific question or problem when prompted with the task-prompt concatenated to the question. The set of units of evolution (e.g. 50). The informational structure that is being evolved, here consisting of a task-prompt set (typically 2), a mutation-prompt, and in the few-shot case a set of 2-3 contexts (workings out). ## Appendix B A Typical Evolutionary Run The word in context task is one of the 24 instruction induction tasks used in APE. Given two sentences and a homograph word, the LLM must determine whether the homograph word has been used with the same meaning in both sentences. Figure 3 shows an evolutionary run where blue dots are individual fitness evaluations and the red line is the population mean. Over 2000 evaluations, the fitness increases considerably. The best evolved Prompt 1 and Prompt 2 pairs (evaluated on the training set) are shown on the right. ## Appendix C Mutation Prompts ## Appendix D Thinking Styles ## Appendix E Initially Evolved Prompts Example of initial prompts generated by concatenating thinking style with mutation prompt and problem description. ## Appendix F Promptbreeder as Self-Referential Self-Improvement System Why is Promptbreeder self-referential, i.e., in what way does some part (e.g. a prompt) causally influence (encode, and potentially improve) itself by a process which is dependent on its own state? Promptbreeder has several pathways that facilitate this self-referential improvement: (i) Initial prompts are a function of the LLM parameters (Initialization Phase). (ii) Initial mutation prompts are a function of the LLM parameters (Initialization Phase). (iii) Offspring prompts are a function of the initial prompts, the initial mutation prompts, and the LLM parameters (Direct Mutation and Estimation of Distribution Mutation). (iv) Offspring mutation prompts are a function of initial mutation prompts and the LLM parameters (Hyper Mutation). (v) The working out for an answer is a function of prompts and the LLM parameters (Inference). (vi) Offspring prompts can be a function of the workings out of an answer and the LLM parameters (Lamarckian Mutation). Figure 2 shows increasingly complex self-referential causal structures influencing prompt generation. LLMs already encode knowledge about a vast array of problems. With this in mind, Promptbreeder can be seen as a mechanism to extract this knowledge through a diversity of causal processes that generate prompt strategies as well as mutation prompts used to create variations of prompt strategies, which in turn influence the the workings out generated by the LLM at inference time . Consequently, these workings out can influence prompt strategies via Lamarckian mutation. The richer the set of pathways to facilitate this, the more self-referential the LLMs interaction with itself is. This allows the LLM to influence how it works by extracting further information from itself and distilling this into a prompt or mutation prompt, which it shows again to itself for further refinement. There are several pathologies that could arise from such self-referential processes of recursive prompting. If the process is unconstrained and uncontrolled then it can diverge (derailment) or get stuck in an attractor. If the output of the LLM is simply fed back into itself with no other context, then we observe these failure cases with higher sampling temperatures favouring escape from attractors. Ideally, we want the LLM to suggest to itself prompt strategies that have maximal relevance for the task at hand and yet permit sufficient ‘thinking outside the box’. It is useful to note a critical aspect in which our algorithm is not self-referential (in a way that thought is): Promptbreeder invents new ways of generating mutants, but it does not invent new (auxiliary) ways of evaluating them (as in Jaderberg et al. (2017b) )—only the externally specified fitness function is used throughout. ## Appendix G Problem Descriptions [SVAMP, SINGLEEQ, ADDSUB, GSM8K, MULTIARITH]: "Solve the math word problem, giving your answer as an arabic numeral." [AQUA-RAT]: "Solve the multiple choice math word problem, choosing (A),(B),(C),(D) or (E)." [ETHOS]: "Determine whether a text contains hate speech." [CSQA]: "Solve the multiple choice math word problem, choosing (A),(B),(C),(D) or (E)." [SQA]: "Work out an answer to the commonsense reasoning question above, and then answer yes or no." ## Appendix H Lamarckian Mutation Example The Lamarckian Prompt components are shown in red . The working out concatenated after the Lamarckian prompt is shown in black, and the continuation (the new prompt) generated by the LLM is shown in blue . ## Appendix I Datasets ### I.1 Control Task-Prompts Here in Table 5 we list the task-prompts used in the controls for Chain-of-thought, Plan and Solve PS, Plan and Solve PS+, Zero-shot APE and OPRO. The zero-shot APE prompt is the one generated to improve over CoT on the MultiArith and GSM8K datasets. ### I.2 Arithmetic Reasoning We evaluate Prompt Evolution using six arithmetic reasoning datasets: (1) GSM8K ( Cobbe et al., 2021 ) is a dataset of 8.5K high quality linguistically diverse grade school math word problems created by human problem writers, (2) SVAMP ( Patel et al., 2021 ) consists of elementary-level short Natural Language state of the world narratives and poses a question about some unknown quantities, (3) MultiArith ( Roy & Roth, 2016 ) benchmark uses math word problems requiring single to multiple operations and steps of reasoning, (4) AddSub ( Hosseini et al., 2014 ) is a dataset of addition- and subtraction-based arithmetic word problems, (5) AQuA-RAT ( Ling et al., 2017 ) (Algebra Question Answering with Rationales) is a dataset that contains algebraic word problems with rationales. (6) SingleEq ( Koncel-Kedziorski et al., 2015 ) dataset comprises grade-school algebra word problems as single equations with varying length which may involve multiple math operations. ### I.3 Commonsense Reasoning For commonsense reasoning we evaluate Prompt Evolution using two datasets: (1) CommonsenseQA ( Talmor et al., 2019 ) is a dataset of multiple-choice questions that require different types of commonsense knowledge to answer correctly. An example question is ”A revolving door is convenient for two direction travel, but it also serves as a security measure at a what? A) bank, B) library, C) department store, D) mall, E) new york”; Answer = ”A” (2) StrategyQA ( Geva et al., 2021 ) dataset contains yes/no questions that require multiple steps of reasoning to answer, for example: ”Will the Albany in Georgia reach a hundred thousand occupants before the one in New York?” ### I.4 Hate Speech Classification We experimented with optimizing a long prompt for the hate speech classification task that was attempted in “Automatic Prompt Optimization with “Gradient Descent” and Beam Search” ( Pryzant et al., 2023 ) , which used the ETHOS dataset ( Mollas et al., 2022 ) . Pryzant et al use a working-out-conditioned error detection and error fixing prompt to improve the task specification prompt, a self-referential process similar to our use of the Lamarckian operator. ### I.5 Instruction Induction The Instruction Induction dataset ( Honovich et al., 2023 ) comprises 24 language understanding tasks of varying difficulty, from surface-level spelling and morphosyntactic tasks (e.g., pluralization) to sentence similarity, causality detection, style transfer (e.g., formality) and sentiment analysis. ## Appendix J Example Results ### J.1 ETHOS Evolved Prompt ### J.2 Prompt Evolution Maths results The experimental set up used a population size of 50. The fitness of an individual was its accuracy over a randomly select batch of 100 examples from the training set. Where datasets were not provided with a training/test split (MultiArith, AddSub, SingleEQ and SVAMP) the dataset was split into two equal training and test sets before the experiments were conducted. During experiments the LLM is sampled under three different contexts: Redescriber - generating new prompts; Inducer - generating responses from the question and prompt 1; and Evaluator - generating the final output using prompt 2. The maximum number of tokens sampled under each context was 50, 30 and 5 respectively. The temperature of the Inducer and Evaluator was set to 0.0 in all cases, but the temperature of the Redescriber was initialized from 1.0 to 2.0 and permitted to evolve (like a hyperparameter in population based training). The experiments were run until the training fitness appeared to plateau. At this point the fittest individual from the whole of the evolutionary run was evaluated against the test set. Experiments generally ran for 1-2k fitness evaluations. So that would be 20-40 ’generations’ if a generation is 25 pair evaluations for our populations of 50. Three diversity maintenance methods are used in cases where the system gets trapped on a local optimum: 1) Random character strings (typically of length 50) are appended into the front of the prompt before it is passed into the LLM. 2). Fitness sharing is applied on the basis of BERT similarity between the embeddings of prompts Shir & Bäck (2005) 3. Sampling temperature of the mutant producing LLM (Redescriber) is initialized uniformly from 1.0 to 2.0, and is mutated by addition of a uniform random number in the range -0.2, 0.2 at each replication event. Comparison with PoT, PS and Auto-CoT controls using our model is not provided because PS and PS+ were the best prompts in Plan-and-Solve. ### J.3 Evolved Mutation Prompts ### J.4 Mutation Operator Effectiveness ### J.5 ADDSUB Individual after 1600 mutations. Prompt 0 refers to the first prompt applied to the question to produce a working out. This working out is then concatenated with Prompt 1 to produce the answer. This is the same as in Plan-And-Solve. We find that in the few-shot evolution case, the contexts dominate, and often the task-prompts drift into nonsense. They are less critically determining of fitness than the evolved contexts. ### J.6 AQUA Individual after 1400 mutations. ### J.7 MULTIARITH Individual after 610 mutations. ### J.8 GSM8K Individual after 1010 mutations. ### J.9 SINGLEEQ Individual after 2010 mutations. ### J.10 SVAMP Individual after 2400 mutations. ## Appendix K APE Instruction Induction tasks To demonstrate Promptbreeder’s ability to evolve few-shot contexts as well as task-prompts we ran few-shot Promptbreeder on all 24 Instruction Induction datasets used in the APE e xperiments. Unlike text-davinci-002 our LLM is not instruction tuned and yet Promptbreeder was able to match or surpass the APE results on 21 out of 24 tasks up to 21%. Three APE controls are provided, see Table 9 . The first two are from previously published results using the text-davinci-002 model. The third modifies our PromptBreeder to use APE’s task-prompt initialisation method and then the mutation-prompt from the APE paper “Generate a variation of the following instruction while keeping the semantic meaning” The Instruction Induction datasets we do not start with a problem description so for task-prompt initialisation APE uses induction input examples for each task from the dataset. Instruction inputs are a fixed prompt together a handful of training examples used to infer possible problem descriptions. To compare Promptbreeder to APE, we therefore initialized the task description with a randomly chosen induction input example for each task. The example below is an induction input sample for the ’Larger Animal’ task. I gave a friend an instruction and five inputs. The friend read the instruction and wrote an output for every one of the inputs. Here are the input-output pairs: Input: cougar, flea Output: cougar Input: whale shark, dog Output: whale shark Input: human, bald eagle Output: human Input: flea, great white shark Output: great white shark Input: coyote, tiger Output: tiger The instruction was ### K.1 Best prompts and contexts Here the best few-shot results (evolved prompts and contexts) for the 24 instruction inductions tasks from the APE paper. #### K.1.1 First Letter #### K.1.2 Second Letter #### K.1.3 List Letters #### K.1.4 Starting With #### K.1.5 Pluralization #### K.1.6 Passivization #### K.1.7 Negation #### K.1.8 Antonyms #### K.1.9 Synonyms #### K.1.10 Membership #### K.1.11 Rhymes #### K.1.12 Larger Animal #### K.1.13 Cause Selection #### K.1.14 Formality #### K.1.15 Sum #### K.1.16 Difference #### K.1.17 Number to Word