Self-Consistency Improves Chain of Thought Reasoning in Language Models — part 2/3

Cleaned-text code points [15796, 31132). Bibliography omitted here; source.md includes it.

We perform all experiments in the few-shot setting, without training or fine-tuning the language models. For a fair comparison we use the same prompts as in Wei et al. (2022) : for all arithmetic reasoning tasks we use the same set of 8 manually written exemplars; for each commonsense reasoning task, 4-7 exemplars are randomly chosen from the training set with manually composed chain-of-thought prompts. 6 6 6 Self-consistency is robust to different sets of prompts and we provide a study in Appendix A.1.2 . Full details on the prompts used are given in Appendix A.3 .

##### Sampling scheme.

To sample diverse reasoning paths, we followed similar settings to those suggested in Radford et al. (2019) ; Holtzman et al. (2020) for open-text generation. In particular, for UL2-20B and LaMDA-137B we applied temperature sampling with T = 0.5 T=0.5 and truncated at the top- k k ( k = 40 k=40 ) tokens with the highest probability, for PaLM-540B we applied T = 0.7 , k = 40 T=0.7,k=40 , and for GPT-3 we use T = 0.7 T=0.7 without top- k k truncation. We provide an ablation study in Section 3.5 to show that self-consistency is generally robust to sampling strategies and parameters.

### 3.2 Main Results

We report the results of self-consistency averaged over 10 runs, where we sampled 40 outputs independently from the decoder in each run. The baseline we compare to is chain-of-thought prompting with greedy decoding ( Wei et al., 2022 ) , referred to as CoT-prompting , which has been previously used for decoding in large language models ( Chowdhery et al., 2022 ) .

##### Arithmetic Reasoning

The results are shown in Table 2 . 7 7 7 The standard deviation of self-consistency is ≤ 0.5 \leq 0.5 for all tasks and is thus omitted in the table. Please refer to Figure 2 , Figure 7 and 8 for the standard deviations under varying numbers of sampled paths. Self-consistency improves the arithmetic reasoning performance over all four language models significantly over chain-of-thought prompting. More surprisingly, the gains become more significant when the language model’s scale increases, e.g., we see +3%-6% absolute accuracy improvement over UL2-20B but +9%-23% for LaMDA-137B and GPT-3. For larger models that already achieve high accuracy on most tasks (e.g., GPT-3 and PaLM-540B), self-consistency still contributes significant additional gains with +12%-18% absolute accuracy on tasks like AQuA and GSM8K, and +7%-11% on SVAMP and ASDiv. With self-consistency, we achieve new state-of-the-art results on almost all tasks: despite the fact that self-consistency is unsupervised and task-agnostic, these results compare favorably to existing approaches that require task-specific training, or fine-tuning with thousands of examples (e.g., on GSM8K).

##### Commonsense and Symbolic Reasoning

Table 3 shows the results on commonsense and symbolic reasoning tasks. Similarly, self-consistency yields large gains across all four language models, and obtained SoTA results on 5 out of 6 tasks. For symbolic reasoning, we test the out-of-distribution (OOD) setting where the input prompt contains examples of 2-letters or 2-flips but we test examples of 4-letters and 4-flips (this setting is more challenging as PaLM-540B or GPT-3 can already achieve perfect in-distribution accuracy). In this challenging OOD setting, the gain of self-consistency is still quite significant compared to CoT-prompting with sufficient model sizes.

To show the effect of the number of sampled reasoning paths, we plot the accuracy (mean and standard deviation over 10 runs) with respect to varying numbers of sampled paths (1, 5, 10, 20, 40) in Figure 2 . The results show that sampling a higher number (e.g., 40) of reasoning paths leads to a consistently better performance, further emphasizing the importance of introducing diversity in the reasoning paths. In Table 4 , we show self-consistency yields a richer set of reasoning paths compared to greedy decoding with a few example questions from two tasks.

### 3.3 Self-Consistency Helps When Chain-of-Thought Hurts Performance

Ye & Durrett (2022) show that sometimes chain-of-thought prompting could hurt performance compared to standard prompting in few-shot in-context learning. Here we perform a study using self-consistency to see if it can help fill in the gap, over a set of common NLP tasks, including (1) Closed-Book Question Answering: BoolQ ( Clark et al., 2019 ) , HotpotQA ( Yang et al., 2018 ) , and (2) Natural Language Inference: e-SNLI ( Camburu et al., 2018 ) , ANLI ( Nie et al., 2020 ) and RTE ( Dagan et al., 2005 ; Bar-Haim et al., 2006 ; Giampiccolo et al., 2007 ; Bentivogli et al., 2009 ) .

The results over PaLM-540B are shown in Table 5 . For some tasks (e.g., ANLI-R1, e-SNLI, RTE), adding chain-of-thought does hurt performance compared to standard prompting ( Brown et al., 2020 ) , but self-consistency is able to robustly boost the performance and outperform standard prompting, making it a reliable way to add rationales in few-shot in-context learning for common NLP tasks.

### 3.4 Compare to other existing approaches

We conduct a set of additional studies and show that self-consistency significantly outperforms existing methods including sample-and-rank, beam search, and ensemble-based approaches.

##### Comparison to Sample-and-Rank

One commonly used approach to improve generation quality is sample-and-rank, where multiple sequences are sampled from the decoder and then ranked according to each sequence’s log probability ( Adiwardana et al., 2020 ) . We compare self-consistency with sample-and-rank on GPT-3 code-davinci-001 , by sampling the same number of sequences from the decoder as self-consistency and taking the final answer from the top-ranked sequence. The results are shown in Figure 3 . While sample-and-rank does improve the accuracy with additionally sampled sequences and ranking, the gain is much smaller compared to self-consistency.

##### Comparison to Beam Search

In Table 6 , we compare self-consistency with beam search decoding on the UL2-20B model. For a fair comparison we report the accuracy under the same number of beams and reasoning paths. On both tasks self-consistency outperforms beam search significantly. Note self-consistency can also adopt beam search to decode each reasoning path (results are shown as “Self-consistency using beam search”), but its performance is worse compared to self-consistency with sampling. The reason is that beam search yields a lower diversity in the outputs ( Li & Jurafsky, 2016 ) , while in self-consistency the diversity of the reasoning paths is the key to a better performance.

##### Comparison to Ensemble-based Approaches

We further compare self-consistency to ensemble-based methods for few-shot learning. In particular, we consider ensembling by: (1) prompt order permutation: we randomly permute the exemplars in the prompt 40 times to mitigate model’s sensitivity to prompt order ( Zhao et al., 2021 ; Lu et al., 2021 ) ; and (2) multiple sets of prompts ( Gao et al., 2021 ) : we manually write 3 3 different sets of prompts. We took majority vote of the answers from greedy decoding in both approaches as an ensemble. Table 7 shows that compared to self-consistency, existing ensemble-based approaches achieve a much smaller gain. 8 8 8 Self-consistency is compatible with both ensemble approaches and we show the results in Appendix A.1.4 . In addition, note that self-consistency is different from a typical model-ensemble approach, where multiple models are trained and their outputs are aggregated. Self-consistency acts more like a “self-ensemble” on top of a single language model. We additionally show the results of ensembling multiple models in Appendix A.1.3 where the model-ensembles perform much worse compared to self-consistency.

### 3.5 Additional Studies

We conducted a number of additional experiments to analyze different aspects of the self-consistency method, including its robustness to sampling strategies and parameters, and how it works with imperfect prompts and non-natural-language reasoning paths.

##### Self-Consistency is Robust to Sampling Strategies and Scaling

We show self-consistency is robust to sampling strategies and parameters, by varying T T in temperature sampling ( Ackley et al., 1985 ; Ficler & Goldberg, 2017 ) , k k in top- k k sampling ( Fan et al., 2018 ; Holtzman et al., 2018 ; Radford et al., 2019 ) , and p p in nucleus sampling ( Holtzman et al., 2020 ) , over PaLM-540B in Figure 4 (left). Figure 4 (right) shows that self-consistency robustly improves performance across all scales for the LaMDA-137B model series. The gain is relatively lower for smaller models due to certain abilities (e.g., arithmetic) only emerge when the model reaches a sufficient scale ( Brown et al., 2020 ) .

##### Self-Consistency Improves Robustness to Imperfect Prompts

For few-shot learning with manually constructed prompts, human annotators sometimes make minor mistakes when creating the prompts. We further study if self-consistency can help improve a language model’s robustness to imperfect prompts. 9 9 9 We use the same prompts as before, but swap all the numbers in the reasoning paths with random numbers except the final answer, e.g., from “ There are 3 cars in the parking lot already. 2 more arrive. Now there are 3 + 2 = 5 cars. ” to “ There are 7 cars in the parking lot already. 6 more arrive. Now there are 7 + 6 = 5 cars. ”. We show the results in Table 8 : while imperfect prompts decrease accuracy with greedy decoding (17.1 → \rightarrow 14.9), self-consistency can fill in the gaps and robustly improve the results.

Additionally, we found that the consistency (in terms of % of decodes agreeing with the final aggregated answer) is highly correlated with accuracy (Figure 8 , over GSM8K). This suggests that one can use self-consistency to provide an uncertainty estimate of the model in its generated solutions. In other words, one can use low consistency as an indicator that the model has low confidence; i.e., self-consistency confers some ability for the model to “know when it doesn’t know”.

##### Self-Consistency Works for Non-Natural-Language Reasoning Paths and Zero-shot CoT

We also tested the generality of the self-consistency concept to alternative forms of intermediate reasoning like equations (e.g., from “ There are 3 cars in the parking lot already. 2 more arrive. Now there are 3 + 2 = 5 cars. ” to “ 3 + 2 = 5 ”). The results are shown in Table 8 (“Prompt with equations”): self-consistency still improves accuracy by generating intermediate equations; however, compared to generating natural language reasoning paths, the gain is smaller since the equations are much shorter and less opportunity remains for generating diversity in the decoding process. In addition, we tested self-consistency with zero-shot chain-of-thought ( Kojima et al., 2022 ) and show that self-consistency works for zero-shot CoT as well and improves the results significantly (+26.2%) in Table 8 .

## 4 Related work

##### Reasoning in language models.

Language models are known to struggle in Type 2 tasks, such as arithmetic, logical and commonsense reasoning ( Evans, 2010 ) . Previous work has primarily focused on specialized approaches for improving reasoning ( Andor et al., 2019 ; Ran et al., 2019 ; Geva et al., 2020 ; Piękos et al., 2021 ) . Compared to prior work, self-consistency is applicable to a wide range of reasoning tasks without any additional supervision or fine-tuning, while still substantially improving the performance of the chain-of-thought prompting approach proposed in Wei et al. (2022) .

##### Sampling and re-ranking in language models.

Multiple decoding strategies for language models have been proposed in the literature, e.g., temperature sampling ( Ackley et al., 1985 ; Ficler & Goldberg, 2017 ) , top- k k sampling ( Fan et al., 2018 ; Holtzman et al., 2018 ; Radford et al., 2019 ) , nucleus sampling ( Holtzman et al., 2020 ) , minimum Bayes risk decoding ( Eikema & Aziz, 2020 ; Shi et al., 2022 ) , and typical decoding ( Meister et al., 2022 ) . Other work has sought to explicitly promote diversity in the decoding process ( Batra et al., 2012 ; Li et al., 2016 ; Vijayakumar et al., 2018 ) .

Re-ranking is another common approach to improve generation quality in language models ( Adiwardana et al., 2020 ; Shen et al., 2021 ) . Thoppilan et al. (2022) collect additional human annotations to train a re-ranker for response filtering. Cobbe et al. (2021) train a “verifier” to re-rank generated solutions, which substantially improves the solve rate on math tasks compared to just fine-tuning the language model. Elazar et al. (2021) improve the consistency of factual knowledge extraction by extending pre-training with an additional consistency loss. All these methods require either training an additional re-ranker or collecting additional human annotation, while self-consistency requires no additional training, fine-tuning, nor extra data collection.

##### Extract reasoning paths.

Some previous work has considered task-specific approaches for identifying reasoning paths, such as constructing semantic graphs ( Xu et al., 2021a ) , learning an RNN to retrieve reasoning paths over the Wikipedia graph ( Asai et al., 2020 ) , fine-tuning with human annotated reasoning paths on math problems ( Cobbe et al., 2021 ) , or training an extractor with heuristic-based pseudo reasoning paths ( Chen et al., 2019 ) . More recently, the importance of diversity in the reasoning processes has been noticed, but only leveraged via task-specific training, either through an additional QA model over extracted reasoning paths ( Chen et al., 2019 ) , or by the introduction of latent variables in a commonsense knowledge graph ( Yu et al., 2022 ) . Compared to these approaches, self-consistency is far simpler and requires no additional training. The approach we propose simply couples the generation of reasoning paths and a final answer by sampling from the decoder, using aggregation to recover the most consistent answer without additional modules.

##### Consistency in language models.

Some prior work has shown that language models can suffer from inconsistency in conversation ( Adiwardana et al., 2020 ) , explanation generation ( Camburu et al., 2020 ) , and factual knowledge extraction ( Elazar et al., 2021 ) . Welleck et al. (2020) use “consistency” to refer to generating an infinite-length sequence in recurrent language models. Nye et al. (2021) improve the logical consistency of samples from a System 1 model by adding a System 2-inspired logical reasoning module. In this paper we focus on a slightly different notion of “consistency”, i.e., utilizing answer consistency among diverse reasoning paths to improve accuracy.

## 5 Conclusion and Discussion

We introduced a simple yet effective method called self-consistency, and observed that it significantly improves accuracy in a range of arithmetic and commonsense reasoning tasks, across four large language models with varying scales. Beyond accuracy gains, self-consistency is also useful for collecting rationales when performing reasoning tasks with language models, and for providing uncertainty estimates and improved calibration of language model outputs.