One Embedder, Any Task: Instruction-Finetuned Text Embeddings — part 3/3

Cleaned-text code points [31588, 35862). Bibliography omitted here; source.md includes it.

Semantic textual similarity (STS) tasks evaluate the similarity between two sentences. Given a sentence pair ( t 1 , t 2 ) (t_{1},t_{2}) , the embedding model maps t 1 t_{1} and t 2 t_{2} into embeddings separately, and then the similarity between t 1 t_{1} and t 2 t_{2} is measured by their embedding cosine similarity. The evaluation metric is Spearman’s rank correlation, which measures the correlation between the similarity scores and human judgements.

#### Summarization

Automatic summarization evaluation aims to evaluate the quality of a machine-generated summary given a reference summary. While human evaluations are considered more accurate, automatic evaluations allow for fast, inexpensive development cycles Khashabi et al. (2022) . Given a reference summary r r and a machine-generated summary t t , the embedding model maps them into embeddings separately, and we compute the cosine similarity between r r and t t . Spearman’s rank correlation is reported between human judgements and automatic scores.

### B.2 Prompt Retrieval

Large language models have demonstrated the ability of in-context learning, where the model can perform downstream tasks by conditioning generation on a few task demonstrations Liu et al. (2021) . Su et al. (2022) introduce the prompt retrieval task, where the goal is to retrieve a few in-context learning (i.e., demonstration) examples from annotated examples given a test instance. The embedding model is used to encode all annotated examples and to find the few most similar examples to the test instance based on the cosine similarity. Following Su et al. (2022) , we use the retrieved examples for in-context learning on GPT-J Wang and Komatsuzaki (2021) over 11 diverse downstream tasks (e.g., classification, multiple choice, and text-to-SQL) that are not included in MEDI (thus zero-shot settings). We compare different embedding methods by measuring the average performance on these downstream tasks.

### B.3 Automatic Evaluation for Generation

Similar to summarization evaluation in MTEB, we use the Billboard benchmark Kasai et al. (2022a) to apply InstructOR to automatic evaluations for three additional text generation tasks: MSCOCO image captioning Lin et al. (2014) ; Kasai et al. (2022b) , CNN/DailyMail news summarization Fabbri et al. (2021) , and WMT21 Chinese-to-English translation Barrault et al. (2020) ; Freitag et al. (2021) . Following Kasai et al. (2022a) , we measure the cosine similarity between the generated text and each reference text and take the maximum similarity score over all references available Zhang et al. (2020) . We evaluate all embedding models by the Pearson correlation with the human judgments, again following Kasai et al. (2022a) . We then report the average correlation scores over the three datasets. Note that we do not use the English-to-German dataset in Billboard because our models are trained only on English data.

## Appendix C Full instructions

We list all instructions for each dataset in MEDI in Table 7 and Table 8

## Appendix D Full Results

We provide the detailed evaluation scores in MTEB, Billboard and prompt retrieval benchmarks in Table 9 & 10 .

Ando2005 , augenstein-etal-2016-stance , andrew2007scalable , rasooli-tetrault-2015 , goodman-etal-2016-noise , harper-2014-learning

## Instructions for reporting errors

We are continuing to improve HTML versions of papers, and your feedback helps enhance accessibility and mobile support. To report errors in the HTML that will help us improve conversion and rendering, choose any of the methods listed below:

Click the "Report Issue" ( ) button, located in the page header.

Tip: You can select the relevant text first, to include it in your report.

Our team has already identified the following issues . We appreciate your time reviewing and reporting rendering errors we may not have found yet. Your efforts will help us improve the HTML versions for all readers, because disability should not be a barrier to accessing research. Thank you for your continued support in championing open access for all.

Have a free development cycle? Help support accessibility at arXiv! Our collaborators at LaTeXML maintain a list of packages that need conversion , and welcome developer contributions .