One Embedder, Any Task: Instruction-Finetuned Text Embeddings — part 2/3

Cleaned-text code points [15997, 31588). Bibliography omitted here; source.md includes it.

### 4.1 Instructions Enable Diverse Training

Here we analyze the importance of instructions when training data are diverse. We first split MEDI into symmetric (e.g., text similarity) and asymmetric groups (e.g., open-domain QA), as defined in § 2.3 (see Table § 5 in the appendix for details about the symmetric and asymmetric groups). We then train InstructOR with or without instructions on each group separately.

As shown in Fig. 3 , InstructOR finetuned without instructions yields performance similar to or better than the original GTR model (dotted line), if the data are symmetric or asymmetric only . However, InstructOR suffers if finetuned without task instructions on the combination of both types of data (entire MEDI). In contrast, finetuning with instructions enables the model to benefit from the combination of symmetric and asymmetric data (see that the rightmost bar gets additive performance gains from the asymmetric and symmetric tasks). This result demonstrates the importance of instruction finetuning when diverse data are used for embedding training. Note that training on symmetric tasks only without instructions is similar to Sent-T5. Similarly, training on asymmetric tasks only without instructions is similar to GTR, which is also trained on asymmetric open-domain QA datasets. Departing from these prior methods, instruction-based finetuning enables diverse training on both types.

### 4.2 Instruction Robustness

Previous work Sanh et al. (2022) ; Zhou et al. (2022) shows that instruction-finetuned language models are not robust to paraphrased instructions. Here we measure InstructOR ’s robustness to variation in human-written instructions.

Specifically, we write five paraphrased instructions for all evaluation datasets (Table 6 in Appendix) and measure InstructOR ’s performance gap between the best-performing and the worst-performing instructions. Fig. 4 shows that inclusion of 300 super-NI datasets is critical to the robustness of InstructOR . Removing these datasets from training (w/o super-NI) substantially increases the performance gap between the best- and worst-performing instructions, suggesting that super-NI’s diverse instructions help the model handle different formats and styles.

### 4.3 Complexity of Instructions

Here we further analyze the role of instructions over varying degrees of their complexity. Specifically, we consider four levels of instruction complexity: N/A (no instructions), dataset tags, simple instructions, and detailed instructions (the original instruction format, § 2.3 ). In the dataset tag setup, each example is prepended with its dataset name. For instance, on the Natural Questions dataset, the query is formatted as "Natural Questions; Input: who sings the song Love Story" ). In the simple instruction setup, we use one or two words to describe the domain (e.g., for Natural Questions, the input query is Wikipedia Questions; Input: who sings the song Love Story ). Fig. 5 shows their average performances across all task categories. Even with trivial dataset tags, InstructOR outperforms the original GTR model, illustrating the effectiveness of instructions for diverse training. As more information is provided in the instruction (from tag to simple and from simple to detail), we observe consistent improvements.

### 4.4 Model Sizes and Instruction Finetuning

Fig. 6 studies the influence of model sizes. Specifically, we use GTR-Base (0.1B), GTR-Large (0.3B), and GTR-XL (1.5B). They are pretrained on the same corpus and differ only in the encoder size (the embedding sizes are the same). We compare models of various sizes and report the average performance across all the categories. As the encoder transformer model scales up, the performance continues to increase for both GTR and InstructOR . Nonetheless, the improvement in InstructOR is more pronounced, perhaps because embeddings with instructions benefit from larger capacities. This implies that large models are more generalizable to compute texts in various domains and task types, providing embeddings for general purposes. Further scale-ups are left to future work.

### 4.5 Instructions Mitigate Domain Shifts

One advantage of instruction-based finetuning is that it improves models’ ability to generalize to unseen domains and tasks. To demonstrate this effectiveness, we found three unseen domains that InstructOR was not trained on: geography, biology, and civil comments. As shown in Table 3 , InstructOR largely improves (above the average improvement) GTR-Large’s performance on all three domains, indicating that instructions can help more when applying models to unseen or uncommon domains.

### 4.6 Qualititive Analysis

In this qualitative analysis, we use T-SNE van der Maaten and Hinton (2008) to visualize two example of classification with and without instructions. The desired outcome is, for pairs with the same sentiment to be closer together, and pairs with different sentiment to be farther apart. As shown in Fig. 7 , without instructions, the green dot pairs (different sentiment) are closer together in the embedding space, while the red dot pairs (same sentiment) are farther apart. However, with instructions, our method ( InstructOR ) successfully encodes the red dot pairs into close embeddings and correctly classifies the pairs. The distance between the green dot pairs with different sentiment is also larger in the embedding space with instructions.

## 5 Related Work

#### Text Embeddings

Text embeddings are useful in many applications such as information retrieval Thakur et al. (2021) , text similarity Gao et al. (2021) , prompt retrieval for in-context learning Su et al. (2022) , classification Reimers and Gurevych (2019) , and beyond. Much prior work develops different embedding models for different applications. For example, SBERT Reimers and Gurevych (2019) and SimCSE Gao et al. (2021) are applied solely to text similarity and classification tasks, while DPR Karpukhin et al. (2020) and Contriever Izacard et al. (2022) focus on information retrieval. Different from Sentence-T5 trained only on symmetric data or GTR trained only on asymmetric data, we combine both groups of datasets and build MEDI, which is then used to train InstructOR with instructions. Muennighoff et al. (2022) introduced the massive text embedding benchmark (MTEB), which can be used to evaluate embedding models on a variety of embedding tasks, spanning reranking, classification, information retrieval, bitext mining, pair classification, STS, and summarization. Their benchmark shows that models performing well on one task may not perform well on other tasks. The poor zero-shot transfer abilities of existing embedding models make it difficult to use them in applications where only few labeled data are available. This motivates us to develop a single embedding model that is applicable to a variety of tasks and has better generalization to unseen tasks. Wang et al. (2022a) recently proposed E5, weakly-supervised contrastive pre-trained text embeddings, which achieve strong performance across various tasks on the MTEB benchmark, employing a larger embedding dimension compared to InstructOR .

#### Instruction Finetuning

Recent work demonstrated that instruction-finetuned language models could perform new tasks given a natural language instruction Mishra et al. (2022) ; Zhong et al. (2021) ; Min et al. (2022) ; Sanh et al. (2022) ; Wei et al. (2022) ; Wang et al. (2022b) ; Ouyang et al. (2022) . Nonetheless, instruction finetuning has yet to be studied in the context of broadly-applicable embeddings. In this work, we explore finetuning embedding models to follow human instructions where the instruction specifies eventual use cases. Concurrent work demonstrated that instructions could facilitate information retrieval Asai et al. (2022) , which is related to our InstructOR design. They used instructions to build a task-aware retrieval system and conducted evaluations on the retrieval task; we build a general-purpose embedding model with instructions that can be applied to 8 tasks categories (Fig. 2 ), including retrieval, text similarity, clustering, and text evaluation.

## 6 Conclusion

We introduced InstructOR , a single model that creates broadly-applicable text embeddings using natural language instructions. We constructed MEDI, a collection of diverse datasets, to finetune InstructOR with instructions. Our extensive experiments showed that InstructOR achieves state-of-the-art performance on text embedding benchmarks, as well as prompt retrieval for few-shot in-context learning. We hope that researchers and practitioners will benefit from our embeddings or our datasets for tasks of their interest.

## 7 Limitations

Although InstructOR significantly improves the baseline GTR performance, we were only able to use four negative examples during the model finetuning process due to computation constraints. However, negative examples have been shown to play an important role in contrastive learning Robinson et al. (2021) . We hope that future work will scale up the number of negatives used during finetuning and investigate various methods for mining hard negatives. Additionally, we do not have enough computation resources to apply multitask instruction finetuning to GTR-XXL (4.8B parameters), which is also an area for future exploration.

At the core of InstructOR is the instruction design. While our current unified instruction format has demonstrated effectiveness, future research can explore other instructional elements to further improve performance. For example, previous work Wang et al. (2022b) have shown that incorporating demonstration examples and explanations can be beneficial for instruction-finetuned language models.

## Acknowledgements

We thank Akari Asai, Jack Lin, Minghan Li, and the ARK group at UW for their helpful feedback on this work.

## Appendix A Training Setups

#### Minibatch Sampling

Training is performed on a combination of all training datasets in MEDI. Since the number of examples in each dataset is different in orders of magnitude, we downsample large ones. Details for the downsampled numbers of examples on each dataset are shown in Table 5 in the appendix. At each step, we first randomly select a dataset and then construct a minibatch only using the examples from that dataset. In this way, we ensure that in-batch negatives are sampled from the same dataset, thereby preventing the model from using task differences to predict the negative label. We use the maximum batch size that fits the machine memory and run all our experiments on 40GB A100 GPUs.

#### Training

We initialize InstructOR with the GTR-Large model ( Ni et al., 2021 , 335M parameters) 4 4 4 https://huggingface.co/sentence-transformers/gtr-t5-large . and finetune it on MEDI using the AdamW optimizer with learning rate 2 × 10 − 5 2\times 10^{-5} and warmup ratio 0.1. We use a softmax temperature of 0.01 and finetune InstructOR for 20K steps.

#### Baselines

We use the official MTEB benchmark for comparisons, but here we highlight several strong baselines with the following two types. The first class of baselines is embedding models specializing in information retrieval: Contriever-MS Izacard et al. (2022) , GTR Ni et al. (2021) , and coCondenser-MS Gao and Callan (2022) . They are all trained on open-domain QA datasets such as MS MARCO Bajaj et al. (2016) . The second class of baselines focuses on semantic textual similarity: SimCSE Gao et al. (2021) , Sent-T5 Ni et al. (2022) , and SGPT-NLI Muennighoff (2022) . They are mainly trained on symmetric paraphrase datasets such as NLI Williams et al. (2018) and the Quora question pairs. 5 5 5 https://www.quora.com/q/quoradata/ . All of these baselines are based on pretrained language models, achieving strong performance on the MTEB leaderboard. In particular, Sent-T5-XXL and GTR-XXL (both with 4.8B parameters) achieve the first and second best average performances.

## Appendix B Embedding Evaluations

Here we provide a high-level summary of the evaluation tasks (Table 1 ). Following MTEB Muennighoff et al. (2022) , Billboard Kasai et al. (2022a) , and prompt retrieval Su et al. (2022) , we split 70 evaluation datasets into 9 categories by task objectives. Out of the 70 evaluation tasks, 66 are unseen during training (See Table 5 for datasets included during training), Table 1 for examples and instructions for the evaluation datasets.

### B.1 Massive Text Embedding Benchmark

MTEB Muennighoff et al. (2022) is a comprehensive embedding evaluation benchmark that aims to provide a holistic view of current embedding models’ performance and to discover universal text embeddings applicable to a wide range of tasks. It combines several conventional benchmarks (e.g., BEIR, Thakur et al., 2021 , and STS, Cer et al., 2017 ) and spans a wide range of domain-specific datasets, including science, biology, and medicine. Following Muennighoff et al. (2022) , we also report the average performance over 56 datasets. For each task family, we briefly describe the task objective, evaluation metric, and how embeddings are used.

#### Retrieval

Given a query q q and a corpus D = { p 1 , p 2 ​ … ​ p n } D=\{p_{1},p_{2}...p_{n}\} , retrieval aims to find the most relevant documents p i p_{i} in D D for query q q . The embedding model is used to embed q q and p 1 ​ … ​ p n p_{1}...p_{n} into fixed-sized vectors, and then the similarity between q q and p i p_{i} is measured by their embedding cosine similarity. There are 14 diverse datasets (e.g., Natural Questions, Scifact, and NFCorpus) together with the community question-answering (CQA) benchmark Hoogeveen et al. (2015) . We use NDCG@10 (Normalized Discounted cumulative gain at rank position 10) to measure the performance.

#### Reranking

Reranking ranks a list of documents based on their relevance to a query. Given a query q q and a list of documents D = { p 1 , p 2 ​ … ​ p n } D=\{p_{1},p_{2}...p_{n}\} , the embedding model computes embeddings of both the query and documents, which are then used to rank the documents based on their cosine similarities. We use MAP (mean average precision), a standard metric in reranking, to measure performance.

#### Clustering

The goal of clustering is to group similar documents into meaningful clusters. Given a set of documents, the encoder maps each document into an embedding. The k-means clustering algorithm is then used to partition the embedded documents into clusters. The clustering performance is measured by the v-measure that is independent of the permutations of clustering labels Rosenberg and Hirschberg (2007) .

#### Pair Classification

Pair classification tasks aim to predict a binary label for a pair of texts. An example of this task is paraphrase identification, where the goal is to predict whether two sentences are paraphrases of each other. Given a sentence pair ( t 1 , t 2 ) (t_{1},t_{2}) , the embedding model encodes t 1 t_{1} and t 2 t_{2} separately. The cosine similarity between the two embeddings is then used to predict the label. The average precision score is measured for evaluation.

#### Classification

Classification is a popular way to evaluate the quality of embeddings Conneau and Kiela (2018) . For each example in the classification dataset, the embedding of the input text is used as features to a classifier. The classifier is trained on the training data while sentence embedings are kept frozen. We report the classification accuracy on the test set as the evaluation metric.

#### STS