Smarter, Better, Faster, Longer: A Modern Bidirectional Encoder for Fast, Memory Efficient, and Long Context Finetuning and Inference — part 2/4

Cleaned-text code points [15971, 31916). Bibliography omitted here; source.md includes it.

Context Length Extension After training on 1.7 trillion tokens at a 1024 sequence length and RoPE theta of 10,000, we extend the native context length of ModernBERT to 8192 tokens by increasing the global attention layer’s RoPE theta to 160,000 and train for an additional 300 billion tokens. We first train at a constant lower learning rate 6 6 6 We only lowered the LR for ModernBERT-base, as large already decreased LR during the 1024 token training phase. of 3e-4 for 250 billion tokens on an 8192 token mixture of the original pretraining dataset sampled following Fu et al. (2024) . Next, we upsample higher-quality sources following Gao et al. (2024) and conduct the decay phase with a 1 − sqrt 1-\text{sqrt} LR schedule over 50 billion tokens. This context extension process yielded the most balanced model on downstream tasks, as most of our ablations using only one of these strategies resulted in a performance loss on either retrieval or classification tasks.

## 3 Downstream Evaluation

We performed an extensive set of evaluations, across a large range of tasks, aiming to demonstrate the versatility of ModernBERT in common scenarios.

For all tasks, ModernBERT is evaluated against existing encoders of similar size. The base size, conventionally defined as under 150 million parameters, includes BERT-base Devlin et al. (2019) , DeBERTa-v3-base He et al. (2023) , RoBERTa-base Liu et al. (2019a) , as well as the more recent 8192 context NomicBERT Nussbaum et al. (2024) and GTE-en-MLM-base Zhang et al. (2024) . The large size, conventionally defined as above 300 million and under 500 million parameters, includes BERT-large-uncased Devlin et al. (2019) , DeBERTa-v3-large He et al. (2023) and RoBERTa-large Liu et al. (2019a) and GTE-en-MLM-large Zhang et al. (2024) .

### 3.1 Evaluation Setting

#### 3.1.1 Natural Language Understanding

The General Language Understanding Evaluation (GLUE) benchmark Wang et al. (2018) is the standard Natural Language Understanding (NLU) benchmark for encoder models, aiming to measure how well a model performs across a range of sentence or sentence-pair understanding tasks, such as sentiment detection Liu et al. (2019b) or language entailment, through tasks such as MNLI Williams et al. (2018) . Although GLUE is often regarded as saturated by the best-performing models, such as large language models Zhao et al. (2023) , it remains one of the most commonly used evaluation suites for smaller encoder-based models, and provides a good impression of a model’s performance on common classification tasks Portes et al. (2023) ; Zhang et al. (2024) ; He et al. (2023) .

We follow the practice of previous studies Devlin et al. (2019) ; Liu et al. (2019a) ; He et al. (2023) and conduct a hyperparameter search on each GLUE subset (detailed in Appendix E.1 ) in order to provide values comparable to other models. 7 7 7 As Zhang et al. (2024) do not explicitly mention a parameter sweep, we initially ran the same hyperparameter sweep as we did for ModernBERT, but observed inconsistencies in the results. To avoid under-representing GTE-en-MLM’s capabilities, we choose to use their reported GLUE results.

#### 3.1.2 Text Retrieval

Information Retrieval (IR) is one of the most common applications of encoder-only models, 8 8 8 At the time of this paper’s writing, over half of the 100 most downloaded models on the HuggingFace Model Hub were encoder-based retrieval models. where they are used to represent documents and queries in semantic search Karpukhin et al. (2020) . This domain has recently seen considerable growth and interest following the spread of LLMs where semantic search powered by lightweight models is used to provide relevant context to LLMs as part of Retrieval-Augmented Generation pipelines.

We evaluate models in both the single-vector Dense Passage Retrieval (DPR) Karpukhin et al. (2020) setting and the multi-vector ColBERT Khattab and Zaharia (2020) setting.

We report retrieval results on the popular BEIR evaluation suite Thakur et al. (2021) , the common standard for evaluating retrieval performance across a variety of tasks and domains, using the nDCG@10 metric. For each setting detailed below, we conduct a learning rate sweep based on results over a subset of the BEIR benchmarks to select the final model, detailed in Appendix E.2 .

Single vector retrieval One of the most common approaches to neural retrieval using encoders is DPR Karpukhin et al. (2020) , where a single-vector is used to represent an entire document. The similarity between a query and a document can then be computed through distance operations, such as cosine similarity. Models are finetuned using contrastive learning to create representations which are close if a document is relevant to a query, and distant if not van den Oord et al. (2018) .

We train every base model using the MS-MARCO Bajaj et al. (2016) dataset with mined hard negatives Xuan et al. (2020) on 1.25M samples with a batch size of 16 and learning rate warmup for 5% of the training using sentence-transformers Reimers and Gurevych (2019) .

Multi vector retrieval Multi-vector retrieval, championed by ColBERT Khattab and Zaharia (2020) , seeks to mitigate lost information from compressing an entire sequence into a single vector. In multi-vector retrieval, each document is represented by all of its individual token vectors, and the similarity between a query and a document is computed using the MaxSim 9 9 9 The sum for every query token of its similarity with the most similar document token operator.

We adopt the training setup of JaColBERTv2.5 Clavié (2024) , an update on the ColBERTv2 Santhanam et al. (2022) training procedure, with a batch size of 16 and a 5% learning rate warmup. We train all models by distilling the knowledge of a teacher model by using the KL-Divergence between the normalized teacher and student scores. Models are trained on 810k samples from MS-Marco Bajaj et al. (2016) and teacher scores from BGE-M3 Chen et al. (2024) , using the PyLate library Chaffin and Sourty (2024) .

#### 3.1.3 Long-Context Text Retrieval

With a native 8192 context length, ModernBERT improves long-context performance over most existing encoders. However, there are relatively few standardized long-context benchmarks for encoder-only models, and most benchmarks, such as Needle-in-a-haystack Kamradt (2023) and RULER Hsieh et al. (2024) are geared towards generative tasks. Given this limitation, we demonstrate improved long-context performance on the English subset of MLDR Chen et al. (2024) , a long-context retrieval benchmark comprised of over 200,000 long documents. We evaluate three settings:

Single Vector – Out-Of-Domain Models are trained on short-context MS-MARCO as described above, and is evaluated on long context MLDR without any further fine-tuning.

Single Vector – In Domain Models trained on MS-MARCO are further fine-tuned on long-context MLDR training set before being evaluated.

Multi-Vector – Out-Of-Domain Due to its token-level MaxSim mechanism, ColBERT models are able to generalize to long-context without any specific training Bergum (2024) . We directly evaluate the best checkpoints from Section 3.1.2 without any further fine-tuning on MLDR.

#### 3.1.4 Code Retrieval

Fueled by increasingly good code completion models Jiang et al. (2024a) , downstream applications have quickly grown in popularity following the emergence of code assistants. 10 10 10 Spearheaded by GitHub Copilot in 2021 Encoder-only models are used to process and retrieve large quantities of code-related information under resource constraints, increasing the importance of measuring and improving code capabilities of encoder models Li et al. (2024) . Unlike most previous encoders which were largely trained only on textual data Devlin et al. (2019) ; Liu et al. (2019a) ; Portes et al. (2023) ; Zhang et al. (2024) ; Nussbaum et al. (2024) , ModernBERT is pre-trained on code and uses a code-aware tokenizer 11 11 11 Avoiding issues such as the ones seen in T5 Raffel et al. (2020) , whose vocabulary did not include curly braces. .

To measure programming-related performance, we evaluate all models on CodeSearchNet Husain et al. (2019) , a code-to-text benchmark where the model must identify relevant docstring or comments for code blocks, and StackOverflow-QA Li et al. (2024) , where the model must identify relevant responses to StackOverflow questions, in a "hybrid" setting where documents contain both text and code. The latter benchmark also leverages long-context capabilities, as its queries and documents respectively contain 1,400 and 1,200 words on average, leading to average token counts of over 2000.

We evaluate these benchmarks using the CoIR (CodeIR) framework Li et al. (2024) , as single-vector retrieval tasks. All models are trained by re-using the best hyper-parameters identified in Section 3.1.2 .

### 3.2 Downstream Results and Discussion

Aggregated results for all evaluations are presented in Table 1 . For BEIR and GLUE, the two common evaluation suites, we follow existing practice in reporting the average results. Detailed results are provided in Appendix E .

In terms of downstream performance, ModernBERT is the strongest overall model at both the base and large model sizes. ModernBERT represents a Pareto improvement on all tasks over the original BERT and RoBERTA models, with better performance on every evaluation category.

Short-Context Retrieval On BEIR, both variants of ModernBERT outperform existing encoders in both the DPR and ColBERT settings, including the recent GTE-en-MLM and NomicBERT models designed to serve as better backbones for retrieval Zhang et al. (2024) ; Nussbaum et al. (2024) .

While ModernBERT-base only narrowly edges out GTE-en-MLM-base on DPR evaluations, ModernBERT-large increases its lead despite having comparatively fewer parameters at 395M to GTE-en-MLM-large’s 435M.

Long-Context Retrieval - Single Vector In the DPR setting, ModernBERT achieves impressive performance on MLDR, a long-context text retrieval task. However, these results also highlight an interesting phenomenon: without long-context finetuning ModernBERT outperforms both shorter-context models and the long-context NomicBERT but performs noticeably worse than GTE-en-MLM. The performance gap narrows considerably when evaluated in-domain, with both models performing similarly. This suggests that ModernBERT can effectively process long context sequences as a dense encoder but may require more adapted tuning. We plan to explore multiple potential explanations for this phenomenon in future work, including the impact of local attention or GTE-en-MLM having spent a larger part of its pretraining compute budget on longer sequence lengths Zhang et al. (2024) .

Long-Context Retrieval - Multi-Vector In the ColBERT setting, long-context models (GTE-en-MLM, NomicBERT, and ModernBERT) all outperform short-context models by at least 40 NDCG@10 points without requiring any specific finetuning. These results confirm the findings of Bergum (2024) , who showed that ColBERT models are particularly well-suited to long-context retrieval tasks. Among the long-context models, ModernBERT outperforms other long-context models, with at least a 9 NDCG@10 point lead on both model sizes. We theorize that these sizable gains could be explained by our long pretraining ensuring few, if any, tokens are under-trained, as well as a potentially synergistic effect of local attention with ColBERT-style retrieval, but leave further exploration of this phenomenon to future work.

Natural Language Understanding Both ModernBERT models demonstrate exceptional NLU results, as measured by GLUE. ModernBERT-base surpasses all existing base models, including DeBERTaV3-base, becoming the first MLM-trained model to do so. This is surprising, as DeBERTaV3 was trained with the Replaced-Token-Detection objective, which was previously thought to yield stronger downstream NLU performance Clark et al. (2020) ; He et al. (2023) . ModernBERT-large is the second-best large encoder on GLUE, almost matching DeBERTaV3-large with one-tenth fewer parameters while processing tokens in half the time (see Section 4 ).

Code On programming tasks, in both code-to-text (CodeSearchNet) and longer-context hybrid settings (StackQA), ModernBERT outperforms all other models. This result was expected, as it is the only evaluated encoder to be trained on a data mixture including programming data. These results, combined with ModernBERT’s strong showings on other tasks, indicates that ModernBERT has improved understanding of code at no detriment to its ability to process natural text.

## 4 Efficiency

### 4.1 Evaluation Setting

To measure inference efficiency across multiple sequence lengths, we create 4 synthetic sets of 8192 documents 12 12 12 Many common benchmarks are biased towards low and uniform sequence lengths, which is unrepresentative of many real-world situations. . The first two document sets are fixed-length: in fixed short-context , all documents contain 512 tokens and in fixed long-context all documents contain 8192 tokens 13 13 13 512 being the maximum length of most existing encoders, while 8192 is the maximum length of all long-context ones. . To account for the impact of unpadding, we also create two varying-length document sets, where the number of tokens in each set are defined by a normal distribution centered on half the maximum sequence length, 256 and 4096 tokens, respectively. Full data statistics are provided in Appendix F .

We then evaluate all models based on the number of tokens they can process per second, averaged over ten runs. All efficiency evaluations are ran on a single NVIDIA RTX 4090, one of the target GPUs of ModernBERT outlined in Section 2.1.3 We evaluate the GTE-en-MLM models under two settings: out-of-the box, and with the use of the xformers Lefaudeux et al. (2022) library, which enables efficiency enhancements such as unpadding.

### 4.2 Results

All tokens-per-second efficiency results are presented in Table 2 , with absolute run-times provided in Appendix F . ModernBERT stands out as the most efficient model overall. On short context, it processes fixed-length 512 token inputs faster than all other recent encoders, although slower than the original BERT and RoBERTa models 14 14 14 This is partially due to the relatively low parameter count of BERT and RoBERTa compared to more recent encoders. . On long-context, ModernBERT is faster than all competing encoders, processing documents 2.65 and 3 times faster than the next-fastest encoder at the base and large sizes, respectively. ModernBERT-large’s processing speed at length 8192 (46,801 tokens per second) is closer to that of GTE-en-MLM base (47,507 tokens per second) than it is to GTE-en-MLM-large (16,532 tokens per second).

On variable-length inputs, both GTE-en-MLM and ModernBERT models are considerably faster than all other models, largely due to unpadding. However, ModernBERT remains noticeably more efficient than GTE-en-MLM, processing 14.5-30.9 percent more tokens per second at low context lengths and 98.8-118.8 percent more at longer context lengths, thanks to its use of local attention.

ModernBERT is the overall most memory efficient model on both model sizes. ModernBERT-base is able to process batch sizes twice as large as every other model on both input lengths. ModernBERT-large is slightly less memory efficient than the original BERT-large on short-context inputs, but can process batches at least 60 percent bigger than every other large model.

## 5 Conclusion

We present ModernBERT, an open family of encoder-only models which set a new state of the art over existing encoder models on a wide range of classification and retrieval tasks. We show that encoders benefit from both recent pretraining data scales and architecture improvements from autoregressive LLMs.