One Embedder, Any Task: Instruction-Finetuned Text Embeddings — part 1/3

Cleaned-text code points [0, 15997). Bibliography omitted here; source.md includes it.

# One Embedder, Any Task: Instruction-Finetuned Text Embeddings

###### Abstract

We introduce InstructOR , a new method for computing text embeddings given task instructions: every text input is embedded together with instructions explaining the use case (e.g., task and domain descriptions). Unlike encoders from prior work that are more specialized, InstructOR is a single embedder that can generate text embeddings tailored to different downstream tasks and domains, without any further training. We first annotate instructions for 330 diverse tasks and train InstructOR on this multitask mixture with a contrastive loss. We evaluate InstructOR on 70 embedding evaluation tasks (66 of which are unseen during training), ranging from classification and information retrieval to semantic textual similarity and text generation evaluation. InstructOR , while having an order of magnitude fewer parameters than the previous best model, achieves state-of-the-art performance, with an average improvement of 3.4% compared to the previous best results on the 70 diverse datasets. Our analysis suggests that InstructOR is robust to changes in instructions, and that instruction finetuning mitigates the challenge of training a single model on diverse datasets. Our model, code, and data are available at https://instructor-embedding.github.io .

## 1 Introduction

Text embeddings represent discrete text inputs (e.g., sentences, documents, and code) as fixed-sized vectors that can be used in many downstream tasks. These tasks include semantic textual similarity Agirre et al. (2012) ; Marelli et al. (2014) ; Cer et al. (2017) ; Lin et al. (2018) , information retrieval Mitra et al. (2017) ; Karpukhin et al. (2020) ; Izacard et al. (2022) , automatic text evaluation Zhang et al. (2020) ; Sellam et al. (2020) ; Hessel et al. (2021) , prompt retrieval for in-context learning Liu et al. (2022) ; Rubin et al. (2022) ; Su et al. (2022) , and beyond. Recently, we have seen dramatic advances in learning text embeddings Kiros et al. (2015) ; Conneau et al. (2017) ; Logeswaran and Lee (2018) ; Reimers and Gurevych (2019) ; Gao et al. (2021) ; Ni et al. (2021) ; Ni et al. (2022) that perform well on their intended tasks or datasets.

However, most existing embeddings can have significantly degraded performance when applied to new tasks or domains Thakur et al. (2021) ; Muennighoff et al. (2022) . For example, DPR Karpukhin et al. (2020) is stronger for retrieval than text similarity tasks, and vice versa for SimCSE Gao et al. (2021) . Moreover, existing embeddings usually perform poorly when applied to the same type of task but in different domains such as medicine and finance. A common method to address this issue is to further finetune the embeddings on datasets in downstream tasks and domains, which often requires a lot of annotated data Gururangan et al. (2020) . In this paper, we hypothesize that text embeddings (even for the same text input) can be adjusted to different downstream applications using task and domain descriptions, without further task- or domain-specific finetuning.

We introduce InstructOR ( Instruct ion-based O mnifarious R epresentations), a single multitask model that generates task- and domain-aware embeddings given a text input and its task instructions. It achieves state-of-the-art performance on massively many downstream embedding tasks without any training. At the core of our approach is instruction-based finetuning Zhong et al. (2021) ; Min et al. (2022) ; Sanh et al. (2022) ; Wei et al. (2022) : we embed every input together with its end task and domain instruction, departing from prior approaches to embeddings that only take text input. InstructOR embeds the same input into different vectors for different end goals (e.g., Who sings the song “Love Story”? is embedded into three different vectors for different tasks in Fig. 1 ). As shown in Fig. 2 , InstructOR is trained on MEDI, our new collection of 330 text embedding datasets newly annotated with human-written task instructions (§ 2.3 ). We train InstructOR with a contrastive loss over all datasets that maximizes the similarity between semantically related text pairs while minimizing unrelated pairs.

We extensively evaluate InstructOR on diverse domains (e.g., finance, medicine, and news) and a variety of downstream applications (a total of 70 embedding evaluation datasets, including 66 not seen during training), spanning classification, semantic textual similarity, information retrieval, text generation evaluation, and prompt retrieval for in-context learning. InstructOR significantly outperforms prior state-of-the-art embedding models by an average of 3.4% over the 70 diverse datasets. InstructOR also outperforms a variant that is trained without task instructions (§ 3 ), demonstrating the importance of instructions to create task-aware embeddings. Our analysis shows that instruction finetuning addresses the challenge of training a single model on diverse datasets (§ 4.1 ). Further, we demonstrate that the task diversity of MEDI makes the performance of InstructOR particularly robust to paraphrases in instructions (§ 4.2 ). Overall, these results strongly suggest that instruction finetuning should be adopted broadly for text embeddings, which we support by sharing all of our models and code.

## 2 InstructOR

InstructOR encodes inputs together with task instructions, thereby providing task-specific representations that can be used for many downstream language tasks, without any additional training. Here we introduce the architecture of InstructOR (§ 2.1 ), present how we perform multitask instruction-based finetuning (§ 2.2 ), and describe how we collect and annotate the MEDI training data (§ 2.3 ). By default, we refer "task" to a dataset, and use them interchangeably throughout the paper, while a "task category", such as Retrieval, includes many tasks.

### 2.1 Embedding Architecture

We build InstructOR , based on the single encoder architecture Izacard and Grave (2021) ; Ni et al. (2021) ; Ni et al. (2022) . Following prior work Ni et al. (2021) ; Ni et al. (2022) , we use GTR models as the backbone encoder (GTR-Base for InstructOR -Base, GTR-Large for InstructOR , GTR-XL for InstructOR -XL). The GTR models are initialized from T5 models, pretrained on a web corpus, and finetuned on information search datasets. The availability of different sizes in the GTR model family allows us to explore the scaling behaviors of instruction-finetuned embedding models. Given an input text x x and a task instruction I x I_{x} , InstructOR encodes their concatenation I x ⊕ x I_{x}\oplus x . We then generate a fixed-sized, task-specific embedding E I ​ ( I x , x ) \textbf{E}_{I}(I_{x},x) by applying mean pooling to the last hidden representations over the tokens in x x .

### 2.2 Training Objective

InstructOR is trained by formulating a wide variety of tasks as a text-to-text problem of distinguishing good/bad candidate outputs y ∈ { y + , y i − } y\in\{y^{+},y^{-}_{i}\} given an input x x , where a training sample corresponds to the tuple ( x , I x , y , I y ) (x,I_{x},y,I_{y}) , with I x I_{x} and I y I_{y} being instructions associated with x x and y y , respectively. For example, in a retrieval task, x x is a query, and good/bad y y is a relevant/irrelevant document from some document collection. For a textual similarity task, the input and output have a similar form and typically come from the same source collection. For a classification task, training samples can be formed by choosing y y as text sequences associated with the same vs. different classes for good vs. bad examples (Details about pair construction are in § 2.3 ). The input and output instructions depend on the task. For symmetric tasks such as textual similarity, where the input and output have the same form and encoding objective, the instructions are the same. For asymmetric tasks such as retrieval, where the input is a single sentence query and the output is a document, the instructions reflect that difference.

The goodness of candidate y y for input x x is given by similarity s ⁡ ( x , y ) s(x,y) that is the cosine between their InstructOR embeddings: {align*} s(x, y) = cos( E _I(I_x ⊕x), E _I(I_y ⊕y)) Following Ni et al. (2021) , we maximize the similarity between positive pairs ( x , y + ) (x,y^{+}) and minimize negative pairs { ( x , y i − ) } i = 1 k \{(x,y^{-}_{i})\}_{i=1}^{k} , where k k denotes the number of negative pairs per positive pair. Specifically, our training objective is: {align*} L = e s(x, y + )/ γ ∑ y ∈ B e s(x, y)/ γ , where γ \gamma is the softmax temperature and ℬ \mathcal{B} is a union of ( x , y + ) (x,y^{+}) and { ( x , y i − ) } i = 1 k \{(x,y^{-}_{i})\}_{i=1}^{k} . Further following Ni et al. (2021) , we compute the same loss with x x and y y swapped and add it to the previous loss (i.e., bidirectional in-batch sampled loss).

### 2.3 MEDI: Multitask Embedding Data with Instructions

There are no existing datasets that consist of a variety of tasks for embedding training with instructions. We thus construct a collection of 330 datasets with instructions across diverse task categories and domains: M ultitask E mbeddings D ata with I nstructions (MEDI).

#### Data Construction

We build MEDI by combining 300 datasets from Super-NaturalInstructions (super-NI; Wang et al., 2022b ) with 30 datasets from existing collections designed for embedding training.

The super-NI datasets come with natural language instructions, but positive and negative pairs are not provided. We construct these pairs by using Sentence-T5 embeddings Ni et al. (2022) , 1 1 1 We do not include instruction for Sentence-T5 as it is not fine-tuned with instructions. denoted with E ​ ( ⋅ ) \textbf{E}(\cdot) . For the classification datasets, we calculate the pairwise cosine similarity between examples based on input text embeddings cos ⁡ ( E ​ ( x i ) , E ​ ( x j ) ) \cos(\textbf{E}(x_{i}),\textbf{E}(x_{j})) . An example x i x_{i} with a high similarity to x j x_{j} is used to create a positive pair if both examples have the same class label ( y j + = y i y_{j}^{+}=y_{i} ), and a negative pair if the labels differ ( y j − ≠ y i y_{j}^{-}\neq y_{i} ). For the remaining tasks where the output labels are text sequences, the following scores are first computed: {align*} s_pos = cos( E (x_i), E (x_j))+cos( E (y_i), E (y_j)) {align*} s_neg = cos( E (x_i), E (x_j))-cos( E (y_i), E (y_j)) We select example pairs with the highest s p ​ o ​ s s_{pos} as positive pairs and highest s n ​ e ​ g s_{neg} as hard negative pairs. We use one hard negative together with in-batch sampled negatives in the training. Our later analysis shows that the training data from super-NI particularly improve the instruction robustness in evaluation due to the diverse task definitions (§ 4.2 ).

The other 30 embedding training datasets come from the Sentence Transformers embedding data, 2 2 2 https://huggingface.co/datasets/sentence-transformers/embedding-training-data . KILT Petroni et al. (2021) , and MedMCQA Pal et al. (2022) . These 30 datasets already contain positive pairs; a few of them, such as MSMARCO Bajaj et al. (2016) and Natural Questions Kwiatkowski et al. (2019) , also contain hard negative pairs. Following Ni et al. (2021) , we use four negative pairs (hard or in-batch negatives) during the model finetuning process. Since all of these datasets do not have instructions, we develop a unified instruction template and manually write a specific prompt for each dataset, as described next. 3 3 3 All prompts are reviewed by multiple authors independently to make sure they consistently follow our template. We release these instructions together with our MEDI data.

#### Instruction Annotation

Each training instance from MEDI is a tuple ( x , I x , y , I y ) (x,I_{x},y,I_{y}) , where the natural language instructions I x I_{x} and I y I_{y} describe how the embeddings of x x and y y are used for the task. For example, in open-domain QA (e.g., Natural Questions in Table 1 ), I x I_{x} is “Represent the Wikipedia question for retrieving supporting documents; Input: ,” and I y I_{y} is “Represent the Wikipedia document for retrieval; Input: .”

To make instructions consistent across all datasets in MEDI, we design a unified instruction format that consists of the following parts (see Table 4 in the appendix for instances of each part): • Text Type specifies the type of input text that we encode using the embedding model. For example, for an open-domain QA task, the input type of the query is a question, while the input type of the target is a document.

• Task Objective (Optional) describes the objective of how the input text is used in a task. For example, for a classification task, the task objective is to classify the sentence into some category, while the task objective of the retrieval is to retrieve a relevant document. Because not all sentences are associated with a specific task (e.g., STS targets general encoding), we make this part optional.

• Domain (Optional) describes the task domain. For example, for NewsIR, the domain of the task is news. Because not all tasks specify a domain (e.g., STS deals with general statements),this part is also optional.

The final instruction takes the following format: “ Represent the ( Domain ) Text Type for Task Objective : ." Appendix 8 shows instructions for each dataset in MEDI.

## 3 Experiments

We train InstructOR on the MEDI data and evaluate it on a wide range of 70 downstream tasks. Specifically, we use the MTEB benchmark from recent work Muennighoff et al. (2022) , which consists of 56 datasets over 7 diverse task categories, such as classification, reranking, and information retrieval. We then further apply InstructOR to prompt retrieval for in-context learning and text generation evaluation. In all three settings, InstructOR achieves the state-of-the-art performance. See Appendix § A and § B for our detailed settings.

### 3.1 Main Results

Table 2 presents the results from InstructOR and the baselines over the three benchmarks: MTEB, Billboard, and prompt retrieval. We conduct head-to-head comparison between InstructOR and GTR models with the same size. We also include the performance of other representative models for reference, while they are not meant for direct comparison.

InstructOR achieves the best performance on all three benchmarks on average. Compared to GTR-Large (335M), from which InstructOR is initialized, instruction finetuning enhances the performance by 5.7%, 18.3%, and 5.7% in MTEB, Billboard, and prompt retrieval respectively. Specifically, among all task categories, InstructOR (335M) demonstrates large improvements over GTR-Large on the text evaluation (18.3%), classification (10.1%), and clustering tasks (8.9%). Particularly noteworthy is InstructOR ’s performance compared to the previous state-of-the-art model, Sent-T5-XXL (58.4 vs. 56.5 on average), despite the fact that InstructOR has one order of magnitude fewer parameters (335M vs. 4.8B).

As expected, the retrieval-based models (e.g., GTR-XXL) show strong performance on retrieval and reranking but significantly lag behind on STS and classification. Conversely, similarity-based models (e.g., Sent-T5-XXL) perform well on STS, classification, and text evaluation, but not on retrieval. It suggests that these baselines tend to generate specialized embeddings that only excel at certain tasks, while InstructOR provides universal embeddings that perform well on diverse task categories.

## 4 Analysis and Ablations

We demonstrate InstructOR enables universal text embeddings for many diverse tasks. Here we analyze our results from various perspectives: the importance of instructions (§ 4.1 ), instruction robustness (§ 4.2 ) and complexity (§ 4.3 ), model sizes (§ 4.4 ), domain shifts (§ 4.5 ), and qualitative analysis (§ 4.6 ). By default, we report average performance across all categories.