Promptbreeder: Self-Referential Self-Improvement Via Prompt Evolution — part 1/4

Cleaned-text code points [0, 15991). Bibliography omitted here; source.md includes it.

# Promptbreeder: Self-Referential Self-Improvement via Prompt Evolution

###### Abstract

Popular prompt strategies like Chain-of-Thought Prompting can dramatically improve the reasoning abilities of Large Language Models (LLMs) in various domains. However, such hand-crafted prompt-strategies are often sub-optimal. In this paper, we present Promptbreeder , a general-purpose self-referential self-improvement mechanism that evolves and adapts prompts for a given domain. Driven by an LLM, Promptbreeder mutates a population of task-prompts, evaluates them for fitness on a training set, and repeats this process over multiple generations to evolve task-prompts. Crucially, the mutation of these task-prompts is governed by mutation-prompts that the LLM generates and improves throughout evolution in a self-referential way. That is, Promptbreeder is not just improving task-prompts, but it is also improving the mutation-prompts that improve these task-prompts. Promptbreeder outperforms state-of-the-art prompt strategies such as Chain-of-Thought and Plan-and-Solve Prompting on commonly used arithmetic and commonsense reasoning benchmarks. Furthermore, Promptbreeder is able to evolve intricate task-prompts for the challenging problem of hate speech classification.

## 1 Introduction

Prompting is central to the downstream performance of foundation models. For example, different prompt strategies 1 1 1 See Appendix A for definitions of terminology. can have a significant impact on a model’s reasoning abilities ( Wei et al., 2022 ; Nye et al., 2021 ; Zhou et al., 2022 ; Wang et al., 2022 ; Zhou et al., 2023 ; Wang et al., 2023b ) , multi-modal processing abilities ( Yang et al., 2023b ; Wang et al., 2023d ) , or tool use abilities ( Yao et al., 2022 ; Schick et al., 2023 ) . Furthermore, prompting can improve model distillation ( Wang et al., 2023c ; Hsieh et al., 2023 ) and it can be used to simulate agentic behavior ( Wang et al., 2023a ; Park et al., 2023 ; Wu et al., 2023 ) . However, these prompt strategies are manually engineered. Since the specific way a prompt is phrased can have a dramatic effect on its utility ( Madaan & Yazdanbakhsh, 2022 ) , it raises the question of whether prompt engineering can be automated. Automatic Prompt Engineer ( Zhou et al., 2023 , APE,) attempts to address this by generating an initial distribution of prompts using another prompt that infers the problem from a number of input-output examples from the dataset. However, Zhou et al. found “diminishing returns to further selection rounds as the quality seems to stabilize after three rounds”, and consequently abandoned the use of an iterative APE. We propose a solution to the problem of diminishing returns via a diversity maintaining evolutionary algorithm for self-referential self-improvement of prompts for LLMs.

Schmidhuber (1990) notes that the “program of a neural network is its weight matrix”. Consequently, this “program” can be changed in a self-referential way by the neural network itself ( Schmidhuber, 1993 ; Irie et al., 2022 ) . Such a neural network that improves itself, as well as improving the way it improves itself, might be an important stepping stone towards open-ended self-referential self-improvement of AIs ( Schmidhuber, 2003 ) . However, self-improvement via self-referential weight matrices is costly as it requires additional parameters that modify all of the model’s parameters. Since behaviors and capabilities of LLMs are significantly influenced by the prompts that we provide to them, we can similarly think of prompts as the program of an LLM ( Zhou et al., 2023 ) . In this view, changing a prompt strategy such as the Scratchpad method ( Nye et al., 2021 ) or Chain-of-Thought Prompting ( Wei et al., 2022 ) corresponds to changing the “program” of the LLM. Taking this analogy further, we can use the LLM itself to change its prompts, as well as the way it changes these prompts, moving us towards a fully self-referential self-improving systems grounded in LLMs.

In this paper, we introduce Promptbreeder (PB) for self-referential self-improvement of LLMs. Given a seed set of mutation-prompts (i.e. instructions to modify a task-prompt), thinking-styles (i.e. text descriptions of general cognitive heuristics), and a domain-specific problem description, PB generates variations of the task-prompts and mutation-prompts, exploiting the fact that LLMs can be prompted to act as mutation operators ( Meyerson et al., 2023 ) . Based on the fitness of the evolved task-prompts as measured on the training set, we select a subset of evolutionary units consisting of task-prompts and their associated mutation-prompt, to transmit to future generations. Over multiple generations of PB, we observe prompts adapting to the domain at hand. For example, in a mathematical domain, PB evolved the task-prompt "Show all your working. II. You should use the correct mathematical notation and vocabulary, where appropriate. III. You should write your answer in full sentences and in words. IV. You should use examples to illustrate your points and prove your answers. V. Your workings out should be neat and legible" on GSM8K (see Appendix J ). On a wide range of commonly used benchmarks spanning commonsense reasoning, arithmetic, and ethics, we find that PB outperforms state-of-the-art methods like Chain-of-Thought ( Wei et al., 2022 ) and Plan-and-Solve ( Wang et al., 2023b ) prompting. As PB does not require any parameter updates for self-referential self-improvement, we believe this approach points to an interesting future where larger and more capable LLMs could further amplify the gains of our approach.

In summary, this paper makes the following main contributions: (i) we introduce Promptbreeder, a self-referential self-improvement method for LLMs that evolves prompts for a domain at hand, as well as improves the way it is evolving these prompts, (ii) we report improvements over state-of-the-art prompt strategies on a wide range of commonly used arithemic and commonsense reasoning benchmarks, and (iii) we investigate the various self-referential components of Promptbreeder and their contribution to our results.

## 2 Related Work

Prompting an LLM in the right way is essential to its downstream performance ( Moradi & Samwald, 2021 ; Madaan & Yazdanbakhsh, 2022 ; Zhou et al., 2023 ) . Indeed, even the order in which prompts are presented can heavily influence LLM performance ( Lu et al., 2022 ) . A number of recent works have focused on devising better prompt strategies, or even automating such prompt engineering.

Prompting : Chain-of-Thought Prompting ( Wei et al., 2022 , CoT,) is a popular prompt strategy which provides intermediate reasoning steps as few-shot prompts to an LLM, thereby significantly improving its arithmetic, commonsense, and symbolic reasoning abilities. Notably, the gains of CoT are more pronounced for stronger LLMs. This is intriguing, as it points to the possibility of increasingly capable (and potentially open-ended) self-improving mechanisms on top of adept LLMs—a hypothesis that Promptbreeder directly builds upon. Instead of few-shot CoT prompting, Kojima et al. (2022) demonstrate that LLMs can also be prompted zero-shot (e.g. "Let’s think step by step" ) to produce their own chains of thoughts (Zero-shot CoT) that improve reasoning abilities. Self-Consistency ( Wang et al., 2022 , CoT-SC,) extends CoT by sampling a diverse set of workings out and selecting the most consistent answer. Tree of Thoughts ( Yao et al., 2023 , ToT,) generalizes CoT to multiple workings out that can be expanded or backtracked from. Graph of Thoughts ( Besta et al., 2023 , GoT,) is a further generalization to arbitrary graph structures. Plan-and-Solve Prompting ( Wang et al., 2023b , PS,) encourages an LLM to first devise a plan to solve a problem before attempting to solve it. Similarly, Least-to-Most Prompting ( Zhou et al., 2022 ) encourages an LLM to decompose a problem into subparts, and then to solve each part individually before synthesizing an answer. Self-Refine ( Madaan et al., 2023 ) prompts an LLM to generate a response, to provide feedback on the response, and to finally refine the solution.

In contrast to gradient-free approaches above, Soft Prompting approaches ( Liu et al., 2021 ; Qin & Eisner, 2021 ; Lester et al., 2021 , e.g.,) directly fine-tune continuous prompt representations. Huang et al. (2022) use CoT and CoT-SC on an unlabelled dataset of questions, and subsequently fine-tune an LLM based on generated solutions. Similarly, Zelikman et al. (2022) uses CoT to generate rationales and fine-tunes the LLM based on those examples and rationales that yielded the correct answer. However, as argued by Zhou et al. (2023) , any approach that updates all or a portion of LLM parameters will not scale as models get bigger and, moreover, will not work with the increasing number of LLMs hidden behind an API.

All of the prompt engineering approaches above are domain agnostic but hand designed. Central to our work is the hypothesis that we could do better by employing an automated self-improvement process that can adapt prompts to a domain at hand. Auto-CoT ( Zhang et al., 2023b ) and Automatic-CoT ( Shum et al., 2023 ) automatically find reasoning chains for Few-Shot CoT. Automatic Prompt Engineer ( Zhou et al., 2023 , APE,) uses one generator-prompt to generate prompt candidates, and another mutation-prompt to mutate them. In contrast to APE, our work performs compositional task-specific initialization of mutation-prompts, subsequent online mutation of mutation-prompts, uses special mutation operators that take into account the whole population and elite history, and uses diversity-maintenance methods—all of which help avoid the problem of diminishing returns and diversity loss suffered by APE.

Concurrently to our work, Yang et al. (2023a) developed Optimization by PROmpting (OPRO), a prompt optimization method that varies prompts using a single complex mutation prompt, and evaluates newly generated prompts on a small fixed training set of problems. In contrast, Promptbreeder autonomously evolves multiple LLM generated mutation-prompts as well as task-prompts, and evaluates fitness on random subsets from the whole training set during evolution. At the time of its release, OPRO achieved a score of 80.2% via the optimized zero-shot prompt "Take a deep breath and work on this problem step-by-step" on GSM8K. Promptbreeder surpasses this with 83.9% in the zero-shot setting with the unintuitively simple prompt "SOLUTION"" —further evidence for the sensitivity of LLMs to prompts and the importance on finding effective prompts automatically. Also concurrently to our work, Guo et al. (2023) developed EvoPrompt, which uses a fixed mutation (and crossover) prompt, as well as a prompt that asks for a mutant of the difference between two parent prompts, to produce offspring prompts. EvoPrompt is initialized with a whole population of initial hand-designed task tailored prompts rather than a single problem description as we do. In contrast to the two approaches above, Promptbreeder uses LLMs to self-referentially improve mutation-prompts, and it is able to evolve contexts as well.

Self-Referential Self-Improvement : Developing an open-ended system that can improve itself as well as improving the way it is improving itself ( Schmidhuber, 1993 ; Schmidhuber, 2003 ) is a long-standing open problem in AI research. Schmidhuber (1993) introduced an “introspective” neural network with a self-referential weight matrix that can modify its own weights and, thus, also modify those weights that are governing how its own weights are modified. Recently, Irie et al. (2022) proposed a more scalable self-referential weight matrix taking inspiration from fast weight programmers ( Schmidhuber, 1992 ) . Kirsch & Schmidhuber (2022) propose a self-referential meta-learning approach, combining self-referential weight matrices with ideas from Gödel Machines ( Schmidhuber, 2003 ) , i.e., to allocate more computational resources to better performing solutions. However, since these approaches directly modify parameters of a model, it is unclear how to scale them to the increasing number of parameters in modern LLMs. In contrast, for Promptbreeder the substrate of self-referential self-improvement is natural language, avoiding costly parameter updates altogether.

Open-Endedness and LLMs : Promptbreeder makes use of the observation by Lehman et al. (2022) , Meyerson et al. (2023) and Chen et al. (2023) that LLMs are effective at generating mutations from examples. In addition, LLMs encode human notions of interestingness and can be used to automatically quantify novelty ( Zhang et al., 2023a ) . Promptbreeder is related to Picbreeder ( Secretan et al., 2008 ) , an open-ended human-in-the-loop system that evolves increasingly interesting images. While Picbreeder explores the space of images, Promptbreeder explores the space of prompts and does so without humans in the loop. As Promptbreeder is proposing mutated prompts to itself, it is an example of a system transitioning from “learning from data” to “learning what data to learn from” ( Jiang et al., 2022 ) .

## 3 Promptbreeder

We introduce Promptbreeder, a prompt evolution system that can automatically explore prompts for a given domain and that is able to find task-prompts that improve an LLM’s ability to derive answers to questions in that domain. Promptbreeder is general purpose in that the same system is able to adapt to many different domains.

Promptbreeder makes use of the observation that LLMs can be used to generate variations of input text ( Lehman et al., 2022 ; Meyerson et al., 2023 ; Chen et al., 2023 ) . Figure 1 gives an overview of our method. We are interested in evolving task-prompts. A task-prompt P P is a string used to condition the context of an LLM in advance of some further input Q Q , intended to ensure a better response than if Q Q had been presented in the absence of P P . To evaluate the fitness of each evolved task-prompt, we sample a batch of 100 Q&A pairs from the entire training set of the domain at hand. 2 2 2 Our prompt strategy sequentially applies two task-prompts. The first task-prompt + question produces a continuation. The continuation + second task-prompt produces the final answer.

Promptbreeder generates task-prompts according to an evolutionary algorithm. The mutation operator for this algorithm is itself an LLM, conditioned on a mutation-prompt M M . That is, a mutated task prompt P ′ P^{\prime} is defined by P ′ = LLM ⁡ ( M + P ) P^{\prime}=\LLM(M+P) where ‘ + + ‘ corresponds to string concatenation. A variety of such mutation-prompts are described in Section 3.2 .

Promptbreeder’s main self-referential mechanism stems from applying the evolutionary algorithm not just to task-prompts but also to mutation-prompts. The mutation operator for this meta-level algorithm is again an LLM, now conditioned on a hyper-mutation prompt H H . That is, we obtain a mutated mutation-prompt M ′ M^{\prime} via M ′ = LLM ⁡ ( H + M ) M^{\prime}=\LLM(H+M) .

Given a set of “thinking styles” 𝒯 \mathcal{T} and a set of initial mutation-prompts ℳ \mathcal{M} , as well as a domain-specific problem description D D , Promptbreeder initializes a population of mutated task-prompts (see Section 3.1 ). To clarify, a unit of evolution consists of a set of task-prompts, a mutation-prompt and in the few-shot case, a set of correct workings out (i.e. step-by-step or “chains-of-thought” reasoning steps that led to the correct answer). This means task-prompts and mutation-prompts are in 1:1 correspondence. To evolve this population, we employ a binary tournament genetic algorithm framework ( Harvey, 2011 ) : we sample two individuals from the population, we take the individual with the higher fitness, mutate it (see next section) and overwrite the loser with the mutated copy of the winner.