Demonstrate-Search-Predict: Composing retrieval and language models for knowledge-intensive NLP — part 1/4

Cleaned-text code points [0, 15564). Bibliography omitted here; source.md includes it.

# Demonstrate – Search – Predict : Composing retrieval and language models for knowledge-intensive NLP

###### Abstract

Retrieval-augmented in-context learning has emerged as a powerful approach for addressing knowledge-intensive tasks using frozen language models (LM) and retrieval models (RM). Existing work has combined these in simple “retrieve-then-read” pipelines in which the RM retrieves passages that are inserted into the LM prompt. To begin to fully realize the potential of frozen LMs and RMs, we propose Demonstrate – Search – Predict ( DSP ), a framework that relies on passing natural language texts in sophisticated pipelines between an LM and an RM. DSP can express high-level programs that bootstrap pipeline-aware demonstrations, search for relevant passages, and generate grounded predictions, systematically breaking down problems into small transformations that the LM and RM can handle more reliably. We have written novel DSP programs for answering questions in open-domain, multi-hop, and conversational settings, establishing in early evaluations new state-of-the-art in-context learning results and delivering 37–120%, 8–39%, and 80–290% relative gains against the vanilla LM (GPT-3.5), a standard retrieve-then-read pipeline, and a contemporaneous self-ask pipeline, respectively. We release DSP at https://github.com/stanfordnlp/dsp .

###### Keywords:

## 1 Introduction

In-context learning adapts a frozen language model (LM) to tasks by conditioning the LM on a textual prompt including task instructions and a few demonstrating examples ( McCann et al., 2018 ; Radford et al., 2019 ; Brown et al., 2020 ) . For knowledge-intensive tasks such as question answering, fact checking, and information-seeking dialogue, retrieval models (RM) are increasingly used to augment prompts with relevant information from a large corpus ( Lazaridou et al., 2022 ; Press et al., 2022 ; Khot et al., 2022 ) .

Recent work has shown such retrieval-augmented in-context learning to be effective in simple “retrieve-then-read” pipelines: a query is fed to the RM and the retrieved passages become part of a prompt that provides context for the LM to use in its response. In this work, we argue that the fact that both LMs and RMs consume (and generate or retrieve) natural language texts creates an opportunity for much more sophisticated interactions between them. Fully realizing this would be transformative: frozen LMs and RMs could serve as infrastructure across tasks, enabling ML- and domain-experts alike to rapidly build grounded AI systems at a high level of abstraction and with lower deployment overheads and annotation costs.

Figure 1 begins to illustrate the power of retrieval-augmented in-context learning, but also the limitations of “retrieve-then-read” ( Lazaridou et al., 2022 ; Izacard et al., 2022 ) . Our query is “How many storeys are in the castle David Gregory inherited?” When prompted to answer this, GPT-3.5 ( text-davinci-002 ; Ouyang et al. 2022 ) makes up a fictitious castle with incorrect attributes, highlighting the common observation that knowledge stored in LM parameters is often unreliable ( Shuster et al., 2021 ; Ishii et al., 2022 ) . Introducing an RM component helps, as the LM can ground its responses in retrieved passages, but a rigid retrieve-then-read strategy fails because the RM cannot find passages that directly answer the question.

We introduce the Demonstrate – Search – Predict ( DSP ) framework for in-context learning, which relies entirely on passing natural language text (and scores) between a frozen RM and LM . DSP introduces a number of composable functions that bootstrap training examples ( Demonstrate ), gather information from a knowledge corpus ( Search ), and generate grounded outputs ( Predict ), using them to systematically unify techniques from the retrieval-augmented NLP and the in-context learning literatures ( Lee et al., 2019 ; Khattab et al., 2021a ; Anantha et al., 2020 ; Gao et al., 2022 ; Izacard et al., 2022 ; Dohan et al., 2022 ; Zelikman et al., 2022 ; Zhang et al., 2022 ) . We use DSP to suggest powerful strategies for knowledge-intensive tasks with compositions of these techniques. This reveals new conceptual possibilities for in-context learning in general (§ 2 ), and it allows us to present rich programs that set new state-of-the-art results (§ 3 ).

Figure 1 shows the path that a DSP program might take to arrive at an answer, and Figure 2 illustrates how a deliberate program achieves this. Instead of asking the LM to answer this complex question, the program’s Search stage uses the LM to generate a query “Which castle did David Gregory inherit?” The RM retrieves a passage saying Gregory inherited the Kinnairdy Castle. After a second search “hop” finds the castle’s number of storeys, the Predict stage queries the LM with these passages to answer the original question. Although this program implements behaviors such as query generation, it requires no hand-labeled examples of these intermediate transformations (i.e., of the queries and passages of both retrieval hops). Instead, the Demonstrate stage uses labeled question–answer pairs to implement a form of weak supervision that programmatically annotates the transformations invoked within Search and Predict .

We evaluate several DSP programs on answering questions in open-domain, multi-hop, and conversational settings. In them, we implement novel and reusable transformations such as bootstrapping annotations for all of our pipelines with weak supervision (§ 2.3 ), reliably rewriting questions to resolve conversational dependencies and iteratively decompose complex queries with summarization of intermediate hops (§ 2.4 ), and generating grounded responses from multiple passages with self-consistency (§ 2.5 ). We report preliminary results on Open-SQuAD, HotPotQA, and QReCC using the frozen LM GPT-3.5 and RM ColBERTv2 ( Khattab & Zaharia, 2020 ; Santhanam et al., 2022b ) with no fine-tuning. Our DSP programs deliver 37–120%, 8–39%, and 80–290% relative gains against corresponding vanilla LMs, a standard retrieve-then-read pipeline, and a contemporaneous self-ask pipeline ( Press et al., 2022 ) , respectively. Future versions of this report will include additional test tasks and LM choices.

In summary, this work makes the following contributions. First, we argue that simple task-agnostic pipelines for in-context learning should give way to deliberate, task-aware strategies. Second, we show that this shift need not be a burden: with DSP , such strategies can be easily expressed as short programs using composable operators. Third, this composability spawns powerful capacities, like automatically annotating demonstrations for complex pipelines from end-task labels. Fourth, for three knowledge-intensive tasks, we implement rich programs that establish state-of-the-art results for in-context learning.

## 2 Demonstrate – Search – Predict

We now introduce the DSP framework and show its expressive power by suggesting a number of strategies in which the LM and RM can come together to tackle complex problems effectively. We show in § 3 that such strategies outperform existing in-context learning methods. We begin by discussing the LM and RM foundation modules on which DSP is built (§ 2.1 ) and then the datatypes and control flow within DSP (§ 2.2 ). Subsequently, we discuss each of the three inference stages: Demonstrate (§ 2.3 ), Search (§ 2.4 ), and Predict (§ 2.5 ).

### 2.1 Pretrained Modules: LM and RM

A DSP program defines the communication between the language model LM and the retrieval model RM .

#### Language Model

We invoke a frozen language model LM to conditionally generate (or score ) text. For each invocation, the program prepares a prompt that adapts the LM to a specific function (e.g., answering questions or generating queries). A prompt often includes instructions, a few demonstrations of the desired behavior, and an input query to be answered.

As in Figure 2 , the LM generates not only: (i) the final answer to the input question (in the Predict stage), but also (ii) intermediate “hop” queries to find useful information for the input question ( Search ) as well as (iii) exemplar queries that illustrate how to produce queries for questions in the training set ( Demonstrate ). This systematic use of the LM is a hallmark of DSP programs.

#### Retrieval Model

DSP programs also invoke a frozen retrieval model RM to retrieve the top- k k most “relevant” text sequences for a given query . The RM can index a massive set of pre-defined passages for scalable search, and those passages can be updated without changing the retrieval parameters. The RM accepts free-form textual inputs and specializes in estimating the relevance (or similarity) of a text sequence to a query.

As in Figure 2 , the RM is responsible for retrieving (i) passages for each query generated by the LM (during the Search stage), but also (ii) passages that are used within demonstrations ( Demonstrate ). In the latter case, the RM ’s contributions are less about providing directly relevant information to the input question and more about helping the LM adapt to the domain and task.

Though not utilized in this example, the RM is also used in DSP for functions like retrieving “nearest-neighbor” demonstrations from task training data ( Demonstrate ) and selecting well-grounded generated sequences from the LM ( Predict ).

### 2.2 Datatypes and Control Flow

We have implemented the DSP framework in Python. The present section introduces the core data types and composable functions provided by the framework. We use illustrative code snippets to ground the examples, and to convey the power that comes from being able to express complex interactions between the LM and RM in simple programs.

#### The Example Datatype

To conduct a task, a DSP program manipulates one or more instances of the Example datatype. An Example behaves like a Python dictionary with multiple fields. The program is typically provided with a few training examples. The code snippet below illustrates this for multi-hop question answering.

⬇

This snippet contains two labeled examples, each with a multi-hop question (e.g., “In which city did Akeem Ellis play in 2017?”) and its short answer (“Ellesmere Port”). Arbitrary keys and values are allowed within an Example , though typical values are strings or lists of strings.

In this task, we are unlikely to find an individual passage that provides the answer to any question. For example, the first training example can probably be resolved only by first answering the question of who discovered Palomar (“Edwin Hubble”) and then addressing the question of Hubble’s birth date using different evidence passages. We typically assume that the human-labeled training data do not include labels for intermediate transformations (e.g., queries for individual hops) that would be useful for following these steps, and so it is the job of the DSP program to discover these strategies via in-context learning.

#### A DSP Program

The following code snippet is a complete program for resolving multi-hop questions like those in Figure 1 , with help from train examples like those above.

⬇

The program takes the input (here, a question) and outputs the system output (its short answer). It starts by creating an Example for the input question and assigning the train field to the training set from the previous snippet. Programs invoke and compose DSP primitives (i.e., built-in functions) to build the Demonstrate , Search , and Predict transformations that define the program.

#### Transformations

A transformation is a function that takes an Example as input and returns an Example , populating new fields (or modifying existing fields) in it. This program invokes three developer-defined transformations, namely, multihop_demonstrate , multihop_search , and multihop_predict . Transformations may themselves invoke other transformations, and they act analogously to layers in standard deep neural network (DNN) programming frameworks such as PyTorch, except that they pass text data instead of tensors between each other and do not involve backpropagation.

We categorize transformations according to their behavior (or purpose) under one of the Demonstrate , Search , and Predict stages. That said, DSP does not impose this categorization and allows us to define functions that may blend these stages. We will discuss each of the three stages next.

### 2.3 Demonstrate

It is known that including examples of the desired behavior from the LM in its prompt typically leads to better performance ( Brown et al., 2020 ) . In DSP , a demonstration is a training example that has been prepared to illustrate specific desired behaviors from the LM . A Demonstrate transformation takes as input x of type Example and prepares a list of demonstrations in x.demos , typically by selecting a subset of the training examples in x.train and bootstrapping new fields in them.

#### Bootstrapping Demonstrations

Examples in the training set typically consist of the input text and the target output of the task. The Demonstrate stage can augment a training example by programmatically bootstrapping annotations for intermediate transformations. In our running “multi-hop” example, the demonstrations illustrate three LM -based transformations: (i) how to break down the input question in order to gather information for answering it (i.e., first-hop retrieval), (ii) how to use information gathered in an earlier “hop” to ask follow-up questions, and (iii) how to use the information gathered to answer complex questions.

⬇

Akin to a specialized map , the annotate primitive accepts a user-defined transformation fn and applies it over a list of training examples. Whenever fn returns an example (rather than None ), annotate caches the intermediate predictions (i.e., the generated queries and retrieved passages). These predictions serve as successful demonstrations for the pipeline transformations. In simple uses, fn may attempt to answer the example “zero-shot” one or more times. This is typically done by invoking the Search and Predict stages of the program. When an answer is produced, if fn assesses it as correct, it returns a populated example in which the intermediate predictions are present.

#### Case Study

The snippet below defines the function multihop_demonstrate , called in Line 3 of multihop_program , and illustrates the usage of annotate .

⬇

In Line 10, multihop_demonstrate invokes annotate , which bootstraps missing fields in training examples by caching annotations from attempt_example . The transformation attempt_example takes a training example d and attempts to answer it in a zero-shot fashion: it creates a copy of d with no demonstrations (Line 4; i.e., zero-shot) and invokes the multi-hop search and predict pipeline (Lines 5 and 6). Each transformation returns an updated version of d with additional fields populated. If the pipeline answers correctly (Line 7), the updated d is returned.

Figure 2 illustrates this behavior. Demonstrate transforms a training question–answer pair to a fully-populated demonstration, including fields such as hop1 and hop2 (i.e., queries for multi-hop search) as well as psg1 and psg2 . When the LM is later invoked to conduct a transformation, say, generating a “second-hop” query during Search , the psg1 field serves as context and the hop2 field serves as a label for this particular training example.

#### Discussion