Smarter, Better, Faster, Longer: A Modern Bidirectional Encoder for Fast, Memory Efficient, and Long Context Finetuning and Inference — part 4/4

Cleaned-text code points [46866, 49894). Bibliography omitted here; source.md includes it.

In the main body, we only report the average score over the 15 very diverse datasets of BEIR. We report the results on every subsets for both single and multi-vector retrieval in Table 7 and Table 8 respectively. For both settings and for every model, we perform a sweep for learning rates in [ 1 ​ e − 5 , 2 ​ e − 5 , 3 ​ e − 5 , 5 ​ e − 5 , 8 ​ e − 5 , 1 ​ e − 4 ] [1e{-5},2e{-5},3e{-5},5e{-5},8e{-5},1e{-4}] and choose the model obtaining the best average result over a subset of datasets composed of NFCorpus, SciFact, TREC-Covid and FiQA as the final model. Best learning rates for every setting are reported in Table 9 . Although ModernBERT showcase strong results across the board, it should be noted that an important factor in its performance is TREC-COVID Voorhees et al. (2021) , potentially showcasing the benefits of ModernBERT being trained with a more recent knowledge cutoff than most existing encoders. However, NomicBERT and GTE have also been trained on updated data, so the cutoff cannot be the only factor affecting the performance.

## Appendix F Efficiency

Full statistics of the synthetic datasets used to evaluate the efficiency of the models in Section 4 are given in Table 10 . The detailed runtimes, alongside with the maximum batch size for every model is detailed in Table 11 .

The high maximum batch-size achieved by ModernBERT models, considerably higher than any other models, highlight the strong memory efficiency of the model at both sizes. Inversely, it is worth noting that while DeBERTaV3 has competitive GLUE performance, it stands out as particularly inefficient, both in its memory use and processing speed. Indeed, on both model sizes, DeBERTaV3’s memory use is 5-to-7 times higher than ModernBERT’s, and it processes inputs, two times slower even in the most favorable scenario where all sequences are at the maximum possible length, thus negating any advantage from unpadding.

## Appendix G Licensing

We release the ModernBERT model architecture, model weights, and training codebase under the Apache 2.0 license.

## Instructions for reporting errors

We are continuing to improve HTML versions of papers, and your feedback helps enhance accessibility and mobile support. To report errors in the HTML that will help us improve conversion and rendering, choose any of the methods listed below:

Click the "Report Issue" ( ) button, located in the page header.

Tip: You can select the relevant text first, to include it in your report.

Our team has already identified the following issues . We appreciate your time reviewing and reporting rendering errors we may not have found yet. Your efforts will help us improve the HTML versions for all readers, because disability should not be a barrier to accessing research. Thank you for your continued support in championing open access for all.

Have a free development cycle? Help support accessibility at arXiv! Our collaborators at LaTeXML maintain a list of packages that need conversion , and welcome developer contributions .