Demonstrate-Search-Predict: Composing retrieval and language models for knowledge-intensive NLP — part 4/4

Cleaned-text code points [47217, 49475). Bibliography omitted here; source.md includes it.

The promise of in-context learning is that we can build complex systems from pretrained components using only natural language as the medium for giving systems instructions and, as we argue for, allowing components to communicate with each other. In this new paradigm, the building blocks are pretrained models and the core operations are natural language instructions and operations on natural language texts. If we can realize this potential, then we can broaden participation in AI system development, rapidly prototype systems for new domains, and maximize the value of specialized pretrained components.

In the current paper, we introduced the Demonstrate – Search – Predict ( DSP ) framework for retrieval augmented in-context learning. DSP consists of a number of simple, composable functions for implementing in-context learning systems as deliberate programs —instead of end-task prompts—for solving knowledge intensive tasks. We implemented DSP as a Python library and used it to write programs for Open-SQuAD, HotPotQA, and QReCC. These programs deliver substantial gains over previous in-context learning approaches. However, beyond any particular performance number, we argue that the central contribution of DSP is in helping to reveal a very large space of conceptual possibilities for in-context learning in general.

## Acknowledgements

We thank Ashwin Paranjape, Amir Ziai, and Rick Battle for valuable discussions and feedback. This work was partially supported by IBM as a founding member of the Stanford Institute for Human-Centered Artificial Intelligence (HAI). This research was supported in part by affiliate members and other supporters of the Stanford DAWN project—Ant Financial, Facebook, Google, and VMware—as well as Cisco, SAP, and the NSF under CAREER grant CNS-1651570. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the authors and do not necessarily reflect the views of the National Science Foundation. We thank Giuseppe Attanasio for his public L a T e X GitHub-style Python code formatting gist. 4 4 4 https://gist.github.com/g8a9/07c2be12ae02cfad4aa430d77dc940cb We also thank Riley Goodside for his public tips on formatting LM prompts (at @goodside on Twitter).