Self-Refine: Iterative Refinement with Self-Feedback — part 2/5

Cleaned-text code points [15862, 30656). Bibliography omitted here; source.md includes it.

The three main steps of Self-Refine are feedback , refine , and repeating them iteratively. In this section, we perform additional experiments to analyze the importance of each of these steps.

#### The impact of the feedback quality

Feedback quality plays a crucial role in Self-Refine . To quantify its impact, we compare Self-Refine , which utilizes specific, actionable feedback, with two ablations: one using generic feedback and another without feedback (the model may still iteratively refine its generations, but is not explicitly provided feedback to do so). For example, in the Code Optimization task: actionable feedback, such as Avoid repeated calculations in the for loop , pinpoints an issue and suggests a clear improvement. Generic feedback, like Improve the efficiency of the code , lacks this precision and direction. Table 2 shows feedback’s clear influence.

In Code Optimization, performance slightly dips from 27.5 ( Self-Refine feedback) to 26.0 (generic feedback), and further to 24.8 (no feedback). This suggests that while generic feedback offers some guidance – specific, actionable feedback yields superior results.

This effect is more pronounced in tasks like Sentiment Transfer, where changing from our feedback to generic feedback leads to a significant performance drop (43.2 to 31.2), and the task fails without feedback. Similarly, in Acronym Generation, without actionable feedback, performance drops from 56.4 to 48.0, even with iterative refinements. These results highlight the importance of specific, actionable feedback in our approach. Even generic feedback provides some benefit, but the best results are achieved with targeted, constructive feedback.

#### How important are the multiple iterations of feedback - refine ?

Figure 4 demonstrates that on average, the quality of the output improves as the number of iterations increases. For instance, in the Code Optimization task, the initial output ( y 0 y_{0} ) has a score of 22.0, which improves to 28.8 after three iterations ( y 3 y_{3} ). Similarly, in the Sentiment Reversal task, the initial output has a score of 33.9, which increases to 36.8 after three iterations. This trend of improvement is also evident in Constrained Generation, where the score increases from 29.0 to 49.7 after three iterations. Figure 4 highlights the diminishing returns in the improvement as the number of iterations increases. Overall, having multiple feedback - refine iterations significantly enhances the quality of the output, although the marginal improvement naturally decreases with more iterations.

The performance may not always monotonically increase with iterations: in multi-aspect feedback tasks like Acronym Generation, where the output quality can vary during iteration with improvement in one aspect but decline in another aspect. To counter this, Self-Refine generates numerical scores for different quality aspects, leading to a balanced evaluation and appropriate output selection.

#### Can we just generate multiple outputs instead of refining?

Does Self-Refine improve because of the iterative refinement, or just because it generates more outputs? We compare Self-Refine with ChatGPT, when ChatGPT generates k = 4 k=4 samples (but without feedback and refinement). Then, we compare the performance of Self-Refine against these k k initial outputs in a 1 vs. k k evaluation. In other words, we assess whether Self-Refine can outperform all k k initial outputs. The results of this experiment are illustrated in Figure 9 ( Appendix H ). Despite the increased difficulty of the 1 vs. k k setting, the outputs of Self-Refine are still preferred by humans over all k k initial outputs. This shows the importance of refinement according to feedback over the alternative of just generating multiple initial outputs.

#### Does Self-Refine work with weaker models?

The experiments in Section 3.3 were performed with some of the strongest available models; does Self-Refine work with smaller or weaker models as well? To investigate this, we instantiated Self-Refine with Vicuna-13B ( Chiang et al., 2023 ) , a less powerful base model. While Vicuna-13B is capable of generating initial outputs, it struggles significantly with the refinement process. Specifically, Vicuna-13B was not able to consistently generate the feedback in the required format. Furthermore, even when provided with Oracle or hard-coded feedback, it often failed to adhere to the prompts for refinement. Instead of refining its output, Vicuna-13B either repeated the same output or generated a hallucinated conversation, rendering the outputs less effective. We thus hypothesize that since Vicuna-13B was trained on conversations, it does not generalize as well as instruction-based models to test-time few-shot tasks. Example output and analysis is provided in Appendix G .

#### Qualitative Analysis

We conduct a qualitative analysis of the feedback generated by Self-Refine and its subsequent refinements. We manually analyze 70 samples in total (35 success cases and 35 failure cases) for Code Optimization ( Madaan et al., 2023 ) and Math Reasoning ( Cobbe et al., 2021 ) . For both Math Reasoning and Code Optimization, we found that the feedback was predominantly actionable, with the majority identifying problematic aspects of the original generation and suggesting ways to rectify them.

When Self-Refine failed to improve the original generation, the majority of issues were due to erroneous feedback rather than faulty refinements. Specifically, 33% of unsuccessful cases were due to feedback inaccurately pinpointing the error’s location, while 61% were a result of feedback suggesting an inappropriate fix. Only 6% of failures were due to the refiner incorrectly implementing good feedback. These observations highlight the vital role of accurate feedback plays in Self-Refine .

In successful cases, the refiner was guided by accurate and useful feedback to make precise fixes to the original generation in 61% of the cases. Interestingly, the refiner was capable of rectifying issues even when the feedback was partially incorrect, which was the situation in 33% of successful cases. This suggests resilience to sub-optimal feedback. Future research could focus on examining the refiner’s robustness to various types of feedback errors and exploring ways to enhance this resilience. In Figure 5 , we illustrate how Self-Refine significantly improves program efficiency by transforming a brute force approach into a dynamic programming solution, as a result of insightful feedback. Additional analysis on other datasets such as Dialogue Response Generation is provided in Appendix H .

#### Going Beyond Benchmarks

While our evaluation focuses on benchmark tasks, Self-Refine is designed with broader applicability in mind. We explore this in a real-world use case of website generation, where the user provides a high-level goal and Self-Refine assists in iteratively developing the website. Starting from a rudimentary initial design, Self-Refine refines HTML, CSS, and JS to evolve the website in terms of both usability and aesthetics. This demonstrates the potential of Self-Refine in real-world, complex, and creative tasks. See Appendix I for examples and further discussion, including broader, societal impact of our work.

## 5 Related work

Leveraging human- and machine-generated natural language (NL) feedback for refining outputs has been effective for a variety of tasks, including summarization Scheurer et al. (2022) , script generation Tandon et al. (2021) , program synthesis Le et al. (2022a) ; Yasunaga and Liang (2020) , and other tasks Bai et al. (2022a) ; Schick et al. (2022b) ; Saunders et al. (2022a) ; Bai et al. (2022b) ; Welleck et al. (2022) . Refinement methods differ in the source and format of feedback, and the way that a refiner is obtained. Table 3 summarizes some related approaches; see Appendix B for an additional discussion.

#### Source of feedback.

Humans have been an effective source of feedback Tandon et al. (2021) ; Elgohary et al. (2021) ; Tandon et al. (2022) ; Bai et al. (2022a) . Since human feedback is costly, several approaches use a scalar reward function as a surrogate of (or alternative to) human feedback (e.g., Bai et al. (2022a) ; Liu et al. (2022) ; Lu et al. (2022) ; Le et al. (2022a) ; Welleck et al. (2022) ). Alternative sources such as compilers Yasunaga and Liang (2020) or Wikipedia edits Schick et al. (2022b) can provide domain-specific feedback. Recently, LLMs have been used to generate feedback for general domains Fu et al. (2023) ; Peng et al. (2023) ; Yang et al. (2022) , However, ours is the only method that generates feedback using an LLM on its own output, for the purpose of refining with the same LLM .

#### Representation of feedback.

The form of feedback can be generally divided into natural language (NL) and non-NL feedback. Non-NL feedback can come in human-provided example pairs Dasgupta et al. (2019) or scalar rewards Liu et al. (2022) ; Le et al. (2022b) . In this work, we use NL feedback, since this allows the model to easily provide self -feedback using the same LM that generated the output, while leveraging existing pretrained LLMs such as GPT-4.

#### Types of refiners.

Pairs of feedback and refinement have been used to learn supervised refiners Schick et al. (2022b) ; Du et al. (2022) ; Yasunaga and Liang (2020) ; Madaan et al. (2021) . Since gathering supervised data is costly, some methods learn refiners using model generations Welleck et al. (2022) ; Peng et al. (2023) . However, the refiners are trained for each new domain. Finally, Yang et al. (2022) use prompted feedback and refinement specifically tailored for story generation. In this work, we avoid training a separate refiner, and show that the same model can be used as both the refiner and the source of feedback across multiple domains.

#### Non-refinement reinforcement learning (RL) approaches.

Rather than having explicit refinement, an alternative way to incorporate feedback is by optimizing a scalar reward function, e.g. with reinforcement learning (e.g., Stiennon et al. (2020) ; Lu et al. (2022) ; Le et al. (2022a) ). These methods differ from Self-Refine in that the model does not access feedback on an intermediate generation. Second, these RL methods require updating the model’s parameters, unlike Self-Refine .

## 6 Limitations and Discussion

The main limitation of our approach is that the base models need to have sufficient few-shot modeling or instruction-following abilities, in order to learn to provide feedback and to refine in an in-context fashion, without having to train supervised models and rely on supervised data.

Further, the experiments in this work were performed with language models that are not open-sourced, namely GPT-3.5 , ChatGPT, GPT-4 , and Codex . Existing literature ( Ouyang et al., 2022 ) does not fully describe the details of these models, such as the pretraining corpus, model sizes, and model biases. Further, these models are not free to use, and using them for research requires some funding. Nonetheless, we release our code and model outputs to ensure the reproducibility of our work.

Another limitation of our work is that we exclusively experiment with datasets in English. In other languages, the current models may not provide the same benefits.

Finally, there is a possibility for bad actors to use prompting techniques to steer a model to generate more toxic or harmful text. Our approach does not explicitly guard against this.

## 7 Conclusion

We present Self-Refine : a novel approach that allows large language models to iteratively provide self-feedback and refine their own outputs. Self-Refine operates within a single LLM , requiring neither additional training data nor reinforcement learning. We demonstrate the simplicity and ease of use of Self-Refine across a wide variety of tasks. By showcasing the potential of Self-Refine in diverse tasks, our research contributes to the ongoing exploration and development of large language models, with the aim of reducing the cost of human creative processes in real-world settings. We hope that our iterative approach will help drive further research in this area. To this end, we make all our code, data and prompts anonymously available at https://selfrefine.info/ .

## Appendix A Evaluation Tasks

Table 4 lists the tasks in our evaluation, and examples from each task.

## Appendix B Broader Related Work

Compared to a concurrent work, Reflexion Shinn et al. (2023) , our approach involves correction using feedback, whereas their setup involves finding the next best solution in planning using ReAct. While ReAct and Reflexion provide a free-form reflection on whether a step was executed correctly and potential improvements, our approach is more granular and structured, with multi-dimensional feedback and scores. This distinction allows our method to offer more precise and actionable feedback, making it suitable for a wider range of natural language generation tasks, including those that may not necessarily involve step-by-step planning such as open-ended dialogue generation.

#### Comparison with Welleck et al. (2022)

The closest work to ours may be Self-Correction ( Welleck et al., 2022 ) ; however, Self-Correction has several disadvantages compared to Self-Refine : 1. Self-Correction does not train their model to generate explicit feedback; instead, Welleck et al. (2022) trained their models to refine only. As we show in Section 4 and Table 2 , having the model generate explicit feedback results in significantly better refined outputs.

2. Self-Correction trains a separate refiner (or “corrector”) for each task. In contrast, Self-Refine uses instructions and few-shot prompting, and thus does not require training a separate refiner for each task.

3. Empirically, we evaluated Self-Refine using the same base model of GPT-3 as Self-Correction, and with the same settings on the GSM8K benchmark. Self-Correction achieved 45.9% accuracy while Self-Refine (this work) achieved 55.7 % ( ↑ \bm{\uparrow} 9.8 ).

#### Comparison with non-refinement reinforcement learning (RL) approaches.

Rather than having an explicit refinement module, an alternative way to incorporate feedback is by optimizing a scalar reward function, e.g. with reinforcement learning (e.g., Stiennon et al. (2020) ; Lu et al. (2022) ; Le et al. (2022a) ). These methods differ from Self-Refine (and more generally, refinement-based approaches) in that the model cannot access feedback on an intermediate generation. Second, these reinforcement learning methods require updating the model’s parameters, unlike Self-Refine .

See Table 5 for an additional detailed comparison of related work.