Self-Refine: Iterative Refinement with Self-Feedback — part 3/5
Cleaned-text code points [30656, 46394). Bibliography omitted here; source.md includes it.
Method Primary Novelty zero/few shot improvement multi aspect critics NL feedback with error localization iterative framework RLHF Stiennon et al. (2020) optimize for human preference trained on feedback single (human) (not self gen.) Rainier RL Liu et al. (2022) RL to generate knowledge trained on end task single(accuracy) (knowl. only) Quark RL Lu et al. (2022) quantization to edit generations trained on end task single(scalar score) (dense signal) (train time iter.) Code RL Le et al. (2022a) actor critic RL for code improvement trained on end task single(unit tests) (dense signal) DrRepair Yasunaga and Liang (2020) Compiler feedback to iteratively repair trained semi sup. single(compiler msg) (not self gen.) PEER Schick et al. (2022b) doc. edit trained on wiki edits trained on edits single(accuracy) (not self gen.) Self critique Saunders et al. (2022a) few shot critique generation feedback training single(human) (self gen.) Self-correct Welleck et al. (2022) novel training of a corrector trained on end task single (task specific) (limited setting) (limited setting) Const. AI Bai et al. (2022b) train RL4F on automat (critique, revision) pair critique training (fixed set) Self-ask Press et al. (2022) ask followup ques when interim ans correct;final wrong few shot none (none) GPT3 score Fu et al. (2023) GPT can score generations with instruction few shot single(single utility fn) (none) Augmenter Peng et al. (2023) factuality feedback from external KBs few shot single(factuality) (self gen.) Re 3 Yang et al. (2022) ∼ \sim ours: but one domain, trained critics few shot (trained critics) (not self gen.) Self-Refine fewshot iterative multi aspect NL fb few shot multiple(few shot critics) (self gen.)
, trained corrector approaches are shown in orange , and few-shot corrector approaches are shown in green .
## Appendix C Human Evaluation
The A/B evaluation in our study was conducted by the authors, where a human judge was presented with an input, task instruction, and two candidate outputs generated by the baseline method and Self-Refine . The setup was blind, i.e., the judges did not know which outputs were generated by which method. The judge was then asked to select the output that is better aligned with the task instruction. For tasks that involve A/B evaluation, we calculate the relative improvement as the percentage increase in preference rate. The preference rate represents the proportion of times annotators selected the output produced by Self-Refine over the output from the baseline method. Table 6 shows the results.
## Appendix D GPT-4 Evaluation
In light of the impressive achievements of GPT-4 in assessing and providing reasoning for complex tasks, we leverage its abilities for evaluation in Self-Refine . The approach involves presenting tasks to GPT-4 in a structured manner, promoting the model’s deliberation on the task and generating a rationale for its decision. This methodology is demonstrated in Figures 6 , 7 and 8 :
## Appendix E Model Key
We use terminology here: https://platform.openai.com/docs/models/gpt-3-5
## Appendix F Comparison of Self-Refine with State-of-the-art of Few-Shot Learning Models and Fine-Tuned Baselines
In this section, we present a comprehensive comparison of the performance of Self-Refine with other few-shot models and fine-tuned baselines across a range of tasks, including mathematical reasoning and programming tasks. Tables 8 and 7 display the performance of these models on the PIE dataset and GSM tasks, respectively. Our analysis demonstrates the effectiveness of different model architectures and training techniques in tackling complex problems.
## Appendix G Evaluation of Vicuna-13b
We also experiment with Vicuna-13b ( Chiang et al., 2023 ) , a version of LLaMA-13b ( Touvron et al., 2023 ) fine-tuned on conversations sourced from the web. Vicuna-13b was able to consistently follow the task initialization prompt. However, it struggled to follow the prompts intended for feedback and refinement. This often led to outputs that resembled assistant-like responses, a representative example of which can be found in Appendix G .
It’s important to note that we used the same prompts for Vicuna-13b as those used with other models in our study. However, the limited performance of Vicuna-13b suggests that this model may require more extensive prompt-engineering for optimal performance.
#### Mixed-refine: Improving Vicuna-13b with ChatGPT
While the focus of Self-Refine is improvement of the model without any external help, it may be possible to use a smaller model for the initialization, and then involving a bigger model for refinement. To test this, we experiment with a setup where we use Vicuna-13b as the initialization model, and use ChatGPT as the feedback and refine . The results on Math Reasoning show the promise of this approach: while Vicuna-13b was able to get only 24.18% on Math Reasoning, it was able to improve to 40.5% in this mixed-refinement setting.
## Appendix H Additional Analysis
### H.1 Using Oracle Feedback
We experimented with Oracle Feedback following Welleck et al. (2022) . This method uses correctness information to guide model refinement, only progressing to refine stage if the current answer is incorrect. This adjustment notably enhanced performance in the Math Reasoning task, with GPT-3 improving by 4.8% and GPT-4 by 0.7% Table 9 . This indicates the potential of external signals to optimize model performance in particular tasks.
#### Non-monotonic increase in output quality for acronym generation
For tasks with multi-aspect feedback like Acronym Generation, the output quality can fluctuate during the iterative process, improving on one aspect while losing out on another ( Table 10 ). To address this, Self-Refine ’s feedback generates explicit numerical scores to capture the different aspects of output quality. This allows for a more balanced evaluation of outputs and the selection of the most appropriate one. The algorithm selects the best output based on the maximum score across all iterations, as described in Algorithm 1 (line 8). A similar selection is possible for other tasks like Math Reasoning and Sentiment Reversal, while we observe that output quality increases monotonically with iterations.
#### Feedback and Refinement Error Analysis for Response Generation
We perform a detailed error analysis of Self-Refine feedback and refinement process for Dialogue Response Generation, which we summarize in Tables Table 11 and Table 12 .
Table 11 reports the occurrence of different types of errors in our sample, which includes Incorrect Feedback (25%), Generic Feedback (30%), and Incorrect Scoring (10%). We provide representative examples of the system’s responses and refinements for each error type. These errors highlight potential areas for improving our feedback handling mechanism, particularly in the interpretation and understanding of user inputs.
Table 12 breaks down errors found in the refinement stage of Self-Refine . Errors include: not being robust (10%), ignoring feedback (25%), and introducing a new problem (20%). We demonstrate how the model handles a variety of feedback types, how robust it is under different circumstances, and how often it inadvertently introduces new issues. 60% of the times, the model is robust to incorrect or generic feedback. These insights can guide us in enhancing the model’s refinement capabilities, especially in providing accurate and specific responses.
## Appendix I Beyond Benchmarks
Self-Refine demonstrates its iterative feedback and refinement capabilities in the context of website layout generation. ChatGPT initially produces a rudimentary layout for a given topic, and then uses the feedback to suggest specific, actionable improvements, as demonstrated in Figures 10 and 12 . These suggestions range from design changes such as color and font adjustments, to content enhancements and layout modifications. Figures 11 and 13 showcase the final layouts, post-feedback implementation, highlighting the potential and versatility of Self-Refine across different scenarios.
#### Ice Cream Generation
The feedback generated by feedback for ice cream generation: ∙ \bullet Change the background color of the container to a light blue color (#6f2ff).
∙ \bullet Change the font size of the heading to 48px.
∙ \bullet Add a small icon before the "Welcome to our ice cream parlor!" text using the URL https://cdn-icons-png.flaticon.com/512/3622/3622340.png.
∙ \bullet Add an additional paragraph after the existing text with the following text: "We also offer a variety of toppings and cones to complement your ice cream. Visit us today to try our latest flavors and indulge in a sweet treat!"
∙ \bullet Increase the font size of the button text to 24px.
∙ \bullet Update the button color to #9933.
#### Photosynthesis
The feedback generated by feedback for photosynthesis: ∙ \bullet Increase the font size of the text to 18px for better readability.
∙ \bullet Add more information about the benefits of photosynthesis.
∙ \bullet Remove the unnecessary margin-top from the header.
∙ \bullet Add a ruler or divider below the header to separate it from the image.
## Appendix J Statistical Confidence Intervals
Table 13 shows results from Table 1 with Wilson confidence interval Brown et al. (2001) (at α \alpha = 99% confidence interval) and statistical significance. Gains that are statistical significance based on these confidence intervals are marked with an asterisk. We find that nearly all of GPT-4 gains are statistically significant, ChatGPT gains are significant for 4 out of 7 datasets, and GPT-3.5 gains are significant for 3 out of 7 datasets.
## Appendix K New Tasks
#### Constrained Generation
We introduce “CommonGen-Hard," a more challenging extension of the CommonGen dataset Lin et al. (2020) , designed to test state-of-the-art language models’ advanced commonsense reasoning, contextual understanding, and creative problem-solving. CommonGen-Hard requires models to generate coherent sentences incorporating 20-30 concepts, rather than only the 3-5 related concepts given in CommonGen. Self-Refine focuses on iterative creation with introspective feedback, making it suitable for evaluating the effectiveness of language models on the CommonGen-Hard task.
#### Acronym Generation
Acronym generation requires an iterative refinement process to create concise and memorable representations of complex terms or phrases, involving tradeoffs between length, ease of pronunciation, and relevance, and thus serves as a natural testbed for our approach. We source a dataset of 250 acronyms 4 4 4 https://github.com/krishnakt031990/Crawl-Wiki-For-Acronyms/blob/master/AcronymsFile.csv and manually prune it to remove offensive or uninformative acronyms.
## Appendix L Code Readability
Orthogonal to the correctness, readability is another important quality of a piece of code: though not related to the execution results of the code, code readability may significantly affect the usability, upgradability, and ease of maintenance of an entire codebase. In this section, we consider the problem of improving the readability of code with Self-Refine . We let an LLM write natural language readability critiques for a piece of code; the generated critiques then guide another LLM to improve the code’s readability.
### L.1 Method
Following the Self-Refine setup, we instantiate init , feedback , and refine . The init is a no-op — we directly start by critiquing the code with feedback and applying the changes with refine . • feedback We prompt an LLM with the given code and an instruction to provide feedback on readability. We give the LLM the freedom to freely choose the type of enhancements and express them in the form of free text.
• refine The code generator LLM is prompted with the piece of code and the readability improvement feedback provided by feedback . In addition, we also supply an instruction to fix the code using the feedback. We take the generation from the code generator as the product of one iteration in the feedback loop.
Starting from an initial piece of code y 0 y_{0} , we first critique, c 1 = critique ( y 0 ) c_{1}=\text{critique}(y_{0}) , and then edit the code, y 1 = editor ( y 0 , c 1 ) y_{1}=\text{editor}(y_{0},c_{1}) . This is recursively performed N N times, where c k + 1 = critique ( y k ) c_{k+1}=\text{critique}(y_{k}) and y k + 1 = editor ( y k , c k + 1 ) y_{k+1}=\text{editor}(y_{k},c_{k+1}) .
### L.2 Experiments
#### Dataset
We use the CodeNet Puri et al. (2021) dataset of competitive programming. 5 5 5 https://github.com/IBM/Project_CodeNet For our purpose, these are hard-to-read multi-line code snippets. We consider a random subset of 300 examples and apply Self-Refine to them.
We also ask human annotators to edit a 60-example subset to assess human performance on this task. The human annotators are asked to read the code piece and improve its readability.
#### Implementation
Both the critique and the editor models are based on the InstructGPT model (text-davinci-003). We consider the temperature of both T = 0.0 T=0.0 (greedy) and T = 0.7 T=0.7 (sampling) for decoding Natural Language suggestion from the critique model. We always use a temperature T = 0.0 T=0.0 (greedy) when decoding Programming Language from the code editor. Due to budget constraints, we run Self-Refine for N = 5 N=5 iterations. The exact prompts we use can be found in Figures 25 - 26 .
#### Evaluation Methods
We consider a few automatic heuristic-based evaluation metrics, • Meaningful Variable Names: In order to understand the flow of a program, having semantically meaningful variable names can offer much useful information. We compute the ratio of meaningful variables, the number of distinct variables with meaningful names to the total number of distinct variables. We automate the process of extracting distinct variables and the meaningful subset of variables using a few-shot prompted language model.
• Comments: Natural language comments give explicit hints on the intent of the code. We compute the average number of comment pieces per code line.
• Function Units: Long functions are hard to parse. Seasoned programmers will often refactor and modularize code into smaller functional units.
#### Result
For each automatic evaluation metric, the ratio of meaningful variable, of comment, and the number of function units, we compute for each iteration averaged across all test examples and plot for each Self-Refine iteration in 14(a) , 14(b) and 14(c) respectively. The two curves each correspond to critique with temperature T = 0.0 T=0.0 and T = 0.7 T=0.7 . The iteration 0 number is measured from the original input code piece from CodeNet. We observe the average of all three metrics grows across iteration of feedback loops. A diverse generation of a higher temperature in the critique leads to more edits to improve the meaningfulness of variable names and to add comments. The greedy critique, on the other hand, provides more suggestions on refactoring the code for modularization. Figure 15 provides an example of code-readability improving over iterations.
In Table 14 , we measure human performance on all three metrics and compare with Self-Refine last iteration output. At T = 0.7 T=0.7 , Self-Refine produces more meaning variables, more function units and slightly more comments compared to the human annotators on average. At T = 0.0 T=0.0 , Self-Refine produces less meaningful variables, less comments per line but even more function units.
#### Example
Starting Code: v 0 v_{0}
Code v 1 v_{1}
Code v 2 v_{2}
## Appendix M Dialogue Response Generation