Self-Refine: Iterative Refinement with Self-Feedback — part 1/5

Cleaned-text code points [0, 15862). Bibliography omitted here; source.md includes it.

# Self-Refine : Iterative Refinement with Self-Feedback

###### Abstract

Like humans, large language models ( LLM s) do not always generate the best output on their first try. Motivated by how humans refine their written text, we introduce Self-Refine , an approach for improving initial outputs from LLM s through iterative feedback and refinement. The main idea is to generate an initial output using an LLM ; then, the same LLM provides feedback for its output and uses it to refine itself, iteratively. Self-Refine does not require any supervised training data, additional training, or reinforcement learning, and instead uses a single LLM as the generator, refiner and the feedback provider. We evaluate Self-Refine across 7 diverse tasks, ranging from dialog response generation to mathematical reasoning, using state-of-the-art ( GPT-3.5 and GPT-4) LLM s. Across all evaluated tasks, outputs generated with Self-Refine are preferred by humans and automatic metrics over those generated with the same LLM using conventional one-step generation, improving by ∼ \sim 20% absolute on average in task performance. Our work demonstrates that even state-of-the-art LLM s like GPT-4 can be further improved at test-time using our simple, standalone approach. 1 1 1 Code and data at https://selfrefine.info/ .

## 1 Introduction

Although large language models ( LLM s) can generate coherent outputs, they often fall short in addressing intricate requirements. This mostly includes tasks with multifaceted objectives, such as dialogue response generation, or tasks with hard-to-define goals, such as enhancing program readability. In these scenarios, modern LLM s may produce an intelligible initial output, yet may benefit from further iterative refinement—i.e., iteratively mapping a candidate output to an improved one—to ensure that the desired quality is achieved. Iterative refinement typically involves training a refinement model that relies on domain-specific data (e.g., Reid and Neubig (2022) ; Schick et al. (2022a) ; Welleck et al. (2022) ). Other approaches that rely on external supervision or reward models require large training sets or expensive human annotations ( Madaan et al., 2021 ; Ouyang et al., 2022 ) , which may not always be feasible to obtain. These limitations underscore the need for an effective refinement approach that can be applied to various tasks without requiring extensive supervision.

Iterative self -refinement is a fundamental characteristic of human problem-solving ( Simon, 1962 ; Flower and Hayes, 1981 ; Amabile, 1983 ) . Iterative self-refinement is a process that involves creating an initial draft and subsequently refining it based on self-provided feedback. For example, when drafting an email to request a document from a colleague, an individual may initially write a direct request such as “ Send me the data ASAP ”. Upon reflection, however, the writer recognizes the potential impoliteness of the phrasing and revises it to “ Hi Ashley, could you please send me the data at your earliest convenience? ". When writing code, a programmer may implement an initial “quick and dirty” implementation, and then, upon reflection, refactor their code to a solution that is more efficient and readable. In this paper, we demonstrate that LLMs can provide iterative self-refinement without additional training, leading to higher-quality outputs on a wide range of tasks.

We present Self-Refine : an iterative self-refinement algorithm that alternates between two generative steps– feedback and refine . These steps work in tandem to generate high-quality outputs. Given an initial output generated by a model ℳ \mathcal{M} , we pass it back to the same model ℳ \mathcal{M} to get feedback . Then, the feedback is passed back to the same model to refine the previously-generated draft. This process is repeated either for a specified number of iterations or until ℳ \mathcal{M} determines that no further refinement is necessary. We use few-shot prompting ( Brown et al., 2020 ) to guide ℳ \mathcal{M} to both generate feedback and incorporate the feedback into an improved draft. Figure 1 illustrates the high-level idea, that Self-Refine uses the same underlying language model to generate feedback and refine its outputs .

We evaluate Self-Refine on 7 generation tasks that span diverse domains, including natural language and source-code generation. We show that Self-Refine outperforms direct generation from strong LLM s like GPT-3.5 ( OpenAI, ; Ouyang et al., 2022 , text-davinci-003 and gpt-3.5-turbo ; ) and GPT-4 ( OpenAI, 2023 ) by 5-40% absolute improvement. In code-generation tasks, Self-Refine improves the initial generation by up to absolute 13% when applied to strong code models such as Codex ( Chen et al., 2021 , code-davinci-002 ; ) . We release all of our code, which is easily extensible to other LLMs. In essence, our results show that even when an LLM cannot generate an optimal output on its first try, the LLM can often provide useful feedback and improve its own output accordingly. In turn, Self-Refine provides an effective way to obtain better outputs from a single model without any additional training, via iterative (self-)feedback and refinement.

## 2 Iterative Refinement with Self-Refine

Given an input sequence, Self-Refine generates an initial output, provides feedback on the output, and refines the output according to the feedback. Self-Refine iterates between feedback and refinement until a desired condition is met. Self-Refine relies on a suitable language model and three prompts (for initial generation, feedback, and refinement), and does not require training. Self-Refine is shown in Figure 1 and Algorithm 1 . Next, we describe Self-Refine in more detail.

#### Initial generation

Given an input x x , prompt p gen p_{\text{gen}} , and model ℳ \mathcal{M} , Self-Refine generates an initial output y 0 y_{0} : y 0 = ℳ ( p gen ∥ x ) . \displaystyle y_{0}=\mathcal{M}\left(p_{\text{gen}}\|x\right). (1) For example, in Figure 2(d) , the model generates functionally correct code for the given input. Here, p gen p_{\text{gen}} is a task-specific few-shot prompt (or instruction) for an initial generation, and ∥ \| denotes concatenation. The few-shot prompt contains input-output pairs ⟨ x ( k ) , y ( k ) ⟩ \langle x^{(k)},y^{(k)}\rangle for the task. 2 2 2 Few-shot prompting (also referred to as “in-context learning”) provides a model with a prompt consisting of k k in-context examples of the target task, each in the form of input-output pairs ⟨ x i , y i ⟩ \langle x_{i},y_{i}\rangle ( Brown et al., 2020 ) .

#### feedback

Next, Self-Refine uses the same model ℳ \mathcal{M} to provide feedback f ​ b t fb_{t} on its own output, given a task-specific prompt p fb p_{\text{fb}} for generating feedback: f ​ b t = ℳ ⁡ ( p fb ​ ‖ x ‖ ​ y t ) . \displaystyle fb_{t}=\mathcal{M}\left(p_{\text{fb}}\|x\|y_{t}\right). (2)

Intuitively, the feedback may address multiple aspects of the output. For example, in code optimization, the feedback might address the efficiency, readability, and overall quality of the code.

Here, the prompt p fb p_{\text{fb}} provides examples of feedback in the form of input-output-feedback triples ⟨ x ( k ) , y ( k ) , f ​ b ( k ) ⟩ \langle x^{(k)},y^{(k)},fb^{(k)}\rangle . We prompt the model to write feedback that is actionable and specific via f ​ b ( k ) fb^{(k)} . By ‘actionable’, we mean the feedback should contain a concrete action that would likely improve the output. By ‘specific’, we mean the feedback should identify concrete phrases in the output to change. For example, the feedback in Figure 2(e) is “ This code is slow as it uses a for loop which is brute force. A better approach is to use the formula … (n(n+1))/2 ”. This feedback is actionable, since it suggests the action ‘use the formula…’. The feedback is specific since it mentions the ‘for loop’.

#### refine

Next, Self-Refine uses ℳ \mathcal{M} to refine its most recent output, given its own feedback: y t + 1 = ℳ ( p refine ∥ x ∥ y t ∥ f b t ) . \displaystyle y_{t+1}=\mathcal{M}\left(p_{\text{refine}}\|x\|y_{t}\|fb_{t}\right). (3) For example, in Figure 2(f) , given the initial output and the generated feedback, the model generates a re-implementation that is shorter and runs much faster than the initial implementation. The prompt p refine p_{\text{refine}} provides examples of improving the output based on the feedback, in the form of input-output-feedback-refined quadruples ⟨ x ( k ) , y t ( k ) , f ​ b t ( k ) , y t + 1 ( k ) ⟩ \langle x^{(k)},y_{t}^{(k)},fb_{t}^{(k)},y_{t+1}^{(k)}\rangle .

Iterating Self-Refine Self-Refine alternates between feedback and refine steps until a stopping condition is met. The stopping condition stop ⁡ ( f ​ b t , t ) \mathrm{stop}(fb_{t},t) either stops at a specified timestep t t , or extracts a stopping indicator (e.g. a scalar stop score) from the feedback. In practice, the model can be prompted to generate a stopping indicator in p fb p_{\text{fb}} , and the condition is determined per-task.

To inform the model about the previous iterations, we retain the history of previous feedback and outputs by appending them to the prompt. Intuitively, this allows the model to learn from past mistakes and avoid repeating them. More precisely, Equation 3 is in fact instantiated as: y t + 1 = ℳ ⁡ ( p refine ​ ‖ x ‖ ​ y 0 ​ ‖ f ​ b 0 ‖ ​ … ​ ‖ y t ‖ ​ f ​ b t ) . \displaystyle y_{t+1}=\mathcal{M}\left(p_{\text{refine}}\|x\|y_{0}\|fb_{0}\|...\|y_{t}\|fb_{t}\right). (4) Finally, we use the last refinement y t y_{t} as the output of Self-Refine .

Algorithm 1 summarizes Self-Refine , and Figure 2 shows an example of Self-Refine in the Dialogue Response Generation ( Mehri and Eskenazi, 2020 ) and Code Optimization ( Madaan et al., 2023 ) tasks. Appendix S provides examples of the p gen p_{\text{gen}} , p fb p_{\text{fb}} , p refine p_{\text{refine}} prompts for various tasks. The key idea is that Self-Refine uses the same underlying LLM to generate, get feedback, and refine its outputs given its own feedback. It relies only on supervision present in the few-shot examples.

## 3 Evaluation

We evaluate Self-Refine on 7 diverse tasks: Dialogue Response Generation ( Mehri and Eskenazi, 2020 , Appendix M ; ) , Code Optimization ( Madaan et al., 2023 , Appendix N ; ) , Code Readability Improvement ( Puri et al., 2021 , Appendix L ; ) , Math Reasoning ( Cobbe et al., 2021 , Appendix O ; ) , Sentiment Reversal ( Zhang et al., 2015 , Appendix P ; ) , and we introduce two new tasks: Acronym Generation ( Appendix Q ) and Constrained Generation (a harder version of Lin et al. (2020) with 20-30 keyword constraints instead of 3-5; Appendix R )

Examples for all tasks and dataset statistics are provided in Table 4 ( Appendix A ).

### 3.1 Instantiating Self-Refine

We instantiate Self-Refine following the high-level description in Section 2 . The feedback - refine iterations continue until the desired output quality or task-specific criterion is reached, up to a maximum of 4 iterations. To make our evaluation consistent across different models, we implemented both feedback and refine as few-shot prompts even with models that respond well to instructions, such as ChatGPT and GPT-4 .

#### Base LLMs

Our main goal is to evaluate whether we can improve the performance of any strong base LLMs using Self-Refine . Therefore, we compare Self-Refine to the same base LLMs but without feedback-refine iterations. We used three main strong base LLM across all tasks: GPT-3.5 ( text-davinci-003 ), ChatGPT ( gpt-3.5-turbo ), and GPT-4 ( OpenAI, 2023 ) . For code-based tasks, we also experimented with Codex ( code-davinci-002 ). In all tasks, either GPT-3.5 or GPT-4 is the previous state-of-the-art. 3 3 3 A comparison with other few-shot and fine-tuned approaches is provided in Appendix F We used the same prompts from previous work when available (such as for Code Optimization and Math Reasoning); otherwise, we created prompts as detailed in Appendix S . We use greedy decoding with a temperature of 0.7 for all setups.

### 3.2 Metrics

We report three types of metrics: ∙ \bullet Task specific metric: When available, we use automated metrics from prior work (Math Reasoning: % solve rate; Code Optimization: % programs optimized; Constrained Gen: coverage %)

∙ \bullet Human-pref: In Dialogue Response Generation, Code Readability Improvement, Sentiment Reversal, and Acronym Generation, since no automated metrics are available, we perform a blind human A/B evaluation on a subset of the outputs to select the preferred output. Additional details are provided in Appendix C .

∙ \bullet GPT-4 -pref: In addition to human-pref, we use GPT-4 as a proxy for human preference following prior work ( Fu et al., 2023 ; Chiang et al., 2023 ; Geng et al., 2023 ; Sun et al., 2023 ) , and found high correlation (82% for Sentiment Reversal, 68% for Acronym Generation, and 71% for Dialogue Response Generation) with human-pref. For Code Readability Improvement, we prompt GPT-4 to calculate fraction of the variables that are appropriately named given the context (e.g., x = [] → input_buffer = [] \texttt{x = []}\rightarrow\texttt{input\_buffer = []} ). Additional details are provided in Appendix D .

### 3.3 Results

Table 1 shows our main results:

Self-Refine consistently improves over base models across all model sizes, and additionally outperforms the previous state-of-the-art across all tasks. For example, GPT-4 + Self-Refine improves over the base GPT-4 by 8.7% (absolute) in Code Optimization, increasing optimization percentage from 27.3% to 36.0%. Confidence intervals are provided in Appendix J . For code-based tasks, we found similar trends when using Codex ; those results are included in Appendix F .

One of the tasks in which we observe the highest gains compared to the base models is Constrained Generation, where the model is asked to generate a sentence containing up to 30 given concepts. We believe that this task benefits significantly from Self-Refine because there are more opportunities to miss some of the concepts on the first attempt, and thus Self-Refine allows the model to fix these mistakes subsequently. Further, this task has an extremely large number of reasonable outputs, and thus Self-Refine allows to better explore the space of possible outputs.

In preference-based tasks such as Dialogue Response Generation, Sentiment Reversal, and Acronym Generation, Self-Refine leads to especially high gains. For example in Dialogue Response Generation, GPT-4 preference score improve by 49.2% – from 25.4% to 74.6%. Similarly, we see remarkable improvements in the other preference-based tasks across all models.

The modest performance gains in Math Reasoning can be traced back to the inability to accurately identify whether there is any error. In math, errors can be nuanced and sometimes limited to a single line or incorrect operation. Besides, a consistent-looking reasoning chain can deceive LLM s to think that “everything looks good” (e.g., ChatGPT feedback for 94% instances is ’everything looks good’). In Section H.1 , we show that the gains with Self-Refine on Math Reasoning are much bigger (5%+) if an external source can identify if the current math answer is incorrect.

Improvement is consistent across base LLM s sizes Generally, GPT-4 + Self-Refine performs better than GPT-3.5 + Self-Refine and ChatGPT+ Self-Refine across all tasks, even in tasks where the initial base results of GPT-4 were lower than GPT-3.5 or ChatGPT. We thus believe that Self-Refine allows stronger models (such as GPT-4 ) to unlock their full potential, even in cases where this potential is not expressed in the standard, single-pass, output generation. Comparison to additional strong baselines is provided in Appendix F .

## 4 Analysis