Smarter, Better, Faster, Longer: A Modern Bidirectional Encoder for Fast, Memory Efficient, and Long Context Finetuning and Inference — part 3/4

Cleaned-text code points [31916, 46866). Bibliography omitted here; source.md includes it.

ModernBERT has a native sequence length of 8,192 tokens and incorporates recent architecture improvements, such as GeGLU layers, RoPE positional embeddings, and alternating local-global attention. ModernBERT is the first open model to feature entire model unpadding and is the first encoder designed in a hardware-aware way to maximize inference efficiency.

ModernBERT pushes the encoder state of the art forward across a wide range of benchmarks. On GLUE, ModernBERT-base is the first encoder to beat DeBERTaV3-base since its release in 2021. ModernBERT is in a class of its own in code and ColBERT-style long-context retrieval benchmarks, scoring at least 6.85 and 9.1 percentage points higher than the closest model, respectively, while remaining state-of-the-art on short-context retrieval in both single and multi-vector settings.

At the same time, ModernBERT processes short context inputs twice as fast as DeBERTaV3 and long-context inputs two times faster than the next fastest model with best-in-class memory efficiency.

ModernBERT is a generational leap over the original encoder models, with notable performance improvements over BERT and RoBERTa on both classification and retrieval tasks. ModernBERT is one of the few encoders to support long-context and programming applications, while simultaneously setting a new record in encoder inference efficiency.

## 6 Limitations

Language This study focuses exclusively on the English language, and trains on a very large number of tokens. As such, a major limitation of our work is that it is not directly applicable to other languages, and potentially even less-so to lower resources languages.

Biases Our model is trained largely on web data, as a result, all of its representations are subject to the biases present in such data.

Harmful Content Generation The MLM objective gives the model some ability to generate text by suggesting a given token to replace the [MASK] token Samuel (2024) , which could result in the generation of harmful content. However, ModernBERT is not, primarily, a generative model, and as such, has not been trained to and therefore cannot generate longer sequences of text. As a result, it is considerably less likely to be at risk of generating harmful content of any kind.

MLM-only objective Given the strong results of DeBERTav3 on classification tasks but weak ones on retrieval, it seems that a training leveraging both MLM and RTD might be better suited to achieve best results on classification. Extending our work to RTD is thus a promising line of research.

Scaling Besides the architectural modifications, a key aspect of our studies is data scaling. However, other scaling axes, notably in terms of model parameters are left unexplored.

## 7 Acknowledgements

The authors would like to acknowledge & thank the many people who assisted, supported, or offered insights useful for the completion of this project.

We are particularly thankful for the one-off implementation or evaluation work conducted by Jack Cook, Mark Tenenholtz, Johno Whitaker, and Wayde Gilliam. We also extend similar thanks to Zach Nussbaum for assisting in resolving issues we encountered with NomicBERT during evaluation.

We would like to acknowledge Enrico Shippole, Daniel Han, Colin Raffel, Pierre-Carl Langlais, Omar Khattab, Urchade Zaratiana, Aurélien Lac, Amélie Chatelain, and Raphaël Sourty, for their helpful contributions to discussions.

We also thank Weights&Biases for providing free access to their platform, in particular Morgan McGuire and Thomas Capelle for their support.

We thank HuggingFace’s Arthur Zucker, Cyril Vallez, and Pedro Cuenca for assisting with day-one HuggingFace support.

Finally, we acknowledge Orange Business Cloud Avenue as compute provider and their hardware support throughout the project and thank LightOn for sponsoring the compute.

## 8 Contribution Statement

BW, AC, and BC jointly led the project and contributed to all parts of it. BW worked on all aspects of the project and contributed to all major decisions. He led model design, model training, implemented the majority of the model architecture, and assisted with data selection, elevations, and paper writing. AC co-initiated the project and worked on all aspects of it, including project coordination. Notably, he contributed to monitoring training runs and co-led ablations, final evaluations and paper writing. BC initiated the project and worked on all aspects of it. He contributed to model design and co-led final evaluations, led paper writing, and contributed to the context extension data processing. OW led and conducted the majority of the data selection, processing, and discussion, for all stages of training. He also contributed valuable inputs throughout all stages of the project. OH and ST contributed to a majority of the stages of the project, in particular model architecture and training, with both discussions, implementations and paper writing. Other contributions include pretraining monitoring, final traditional evaluations, and ablations. ST specifically worked on adapting the RoPE kernel for unpadded sequences and running the final GLUE benchmarks. OH additionally conducted a thorough investigation into complex issues that arose during training. RB contributed greatly to the initial evaluation work, focusing on ablations and in-training evals. AG and FL contributed to training efficiency, especially in implementing sequence packing. AG and GA contributed to model evaluations, especially in long context evaluations. TA contributed to discussions throughout the project and assisted in integrating the original research implementation with open source software. NC contributed to context extension data mixtures, and provided insight into model training and on improving the quality of code data. IP and JH provided guidance and support throughout the project, especially on key decisions.

## Appendix A Training Settings

Detailed training settings can be found in Table 3 .

During training we used MNLI as a live evaluation, along with validation loss and token accuracy metrics on a 500 million randomly sampled sequences from the source datasets.

We use Composer Mosaic ML Team (2021) as our training framework and optimī Warner (2023) for our optimizer implementations.

### A.1 Batch Size Schedule

Batch size warmup is a common-knowledge trick to speed up model training when working with medium to large batch sizes. Instead of "wasting" a full batch on updating the suboptimal initial weight distribution, we update the model weights on a gradually increasing batch size. Batch size warmup is usually longer than learning rate warmup, and can be thought of as providing a higher initial learning rate with a mini-learning rate decay to the defined learning rate schedule. We warmup ModernBERT’s batch size from 768 to 4,608 over 50 billion tokens and from 448 to 4,928 over 10 billion tokens, for -base and -large, respectively, with an uneven token schedule so each batch size has the same number of update steps.

### A.2 Weight Tiling

Following the Phi family of models Li et al. (2023) ; Javaheripi et al. (2023) , we initialized ModernBERT-large directly from ModernBERT-base’s pretraining weights using center tiling and Gopher layer scaling Rae et al. (2022) . Since Base’s weight matrices are smaller than Large’s, we centered Base’ weights, accounting for each token embedding and attention head, then filled rest the of the weights using wraparound. Like Phi, we tested center initialization with random edge values and tiling from an edge, but both of these underperformed center tiling with wraparound. This weight initialization strategy greatly accelerates ModernBERT-large’s initial training.

### A.3 Weight Decay

We did not apply weight decay to the bias terms or normalization layers. Instead of PyTorch-style decoupled weight decay, we applied fully decoupled weight decay following Loshchilov and Hutter (2019) .

### A.4 Final Checkpoints

Inspired by recent work showing that checkpoint averaging yields stronger final models Dubey et al. (2024) ; Clavié (2024) , we selected our final checkpoints by experimenting with various averaging methods and evaluating them on a subset of evaluation tasks. In no cases did Exponential Moving Average during annealing, as used by Dubey et al. (2024) , result in stronger performance. ModernBERT-base is the result of averaging the 3 best performing annealing checkpoints with the final one. Averaging did not yield successful results on the large size, ModernBERT-Large model is the best performing annealing checkpoint.

## Appendix B Model Design

From Anthony et al. (2024) , in addition to setting attention heads as multiples of 64 and setting the embedding matrix as a power of 2 or multiple of 64, there are three model design choices to maximize performance (assuming float16 or bfloat16 computation):

• Tensor Core Requirement : Weight matrix dimensions should be divisible by 64

• Tile Quantization : Weight matrix is divisible into 128 × 256 blocks.

• Wave Quantization : Number of blocks is divisible by the number of streaming multiprocessors (SM).

Given that we wanted to target good performance across multiple GPUs with a wide variety of SM counts, wave quantization is an impossible ask. So we selected a basket of GPUs (NVIDIA T4, A10, L4, RTX 3090, RTX 4090, A100, and H100) and calculated the approximate SM utilization for each by dividing the modulus blocks by the number of SMs. This appeared to be a decent performance heuristic in our spot checking. We then designed our models to maximize performance on the basket of GPUs, putting more weight on inference GPUs.

## Appendix C Training Log

### C.1 Sampling Issue

Our first pretraining run of ModernBERT-base ended in disaster as the loss exhibited a slow seesaw pattern before slowly diverging. Despite using PyTorch’s distributed random sampler, training metrics suggested that the model was training on the dataset in a non-random order. Like the Olmo authors 15 15 15 We found a comment and GitHub issue about this in the Olmo codebase after resolving the issue ourselves. , we determined that the PyTorch random sampler returns sequentially biased samples when the number of samples is somewhere between 500 million and 1 billion samples 16 16 16 We did not conduct a rigorous statistical analysis to determine exactly when this happens. . We resolved this issue by replacing the PyTorch sampler with NumPy’s PCG64DXSM random sampler.

### C.2 Large Rollback

We rolled back and restarted ModernBERT-large training at a lower learning rate of 5e-5 and lower weight decay of 1e-6 for the last 800 billion tokens. Prior to restarting training, large’s training loss, validation metrics, and live evaluations on MNLI had plateaued for a few hundred billion tokens at the higher 5e-4 learning rate. In contrast, ModernBERT-base showed a continuous, but diminishing, improvement on training loss, validation metrics, and live evaluations through the entire 1.719 trillion token training phase. This highlights one of the risks of training with a constant learning rate, other learning rate schedules can mitigate selecting a too high learning rate (or too small batch size) by lowering the learning rate throughout training.

## Appendix D Architecture ablations

To select the updates to add in the ModernBERT architecture, we performed different ablations, except where stated, most ablations where ran at the 8-20 billion token scale: • We compared two GLU layers, GeGLU and SwiGLU. We find close to no difference between the two and choose to use GeGLU layers.

• Using different percentage of the head dimension for the RoPE dimension (50, 75, 100). Lower percentages gave slightly better results. However, the observed difference was minimal. As the ablations were conducted at a considerably smaller scale than the final training, we choose to err on the side of caution and opt to keep the dimension at 100 % to avoid potentially hindering the capabilities of the fully trained models.

• Both LayerNorm and RMSNorm yielded very similar results. While RMSNorm is theoretically faster, at the time this work was conducted, PyTorch did not have a native RMSNorm implementation, leading to eager-mode RMSNorm being the default implementation used for many users. To ensure ModernBERT has the highest possible out-of-the-box efficiency, we choose to use LayerNorm in the final models.

• We investigated using parallel attention to compute the MLP and attention matrices at the same time, which has been shown to increase processing speeds for larger model sizes Chowdhery et al. (2023) . However, for models within our targe sizes and pre-training sequence length, the speed-up we observed was minimal while we encountered significant degradation in downstream performance. As such, we do not use parallel attention. It is however possible that larger encoders and/or larger sequence lengths might see a different trade-off.

• We explored the use of alternating global/local attention, with global attention every 3 layers and local attention over a 128 token sliding window otherwise. This setup yielded identical downstream performance when compared to the use of global attention in every layer, even at 100 billion tokens, while resulting in major speedups.

• We experimented with multiple tokenizers, before selecting our final one, based on a modified OLMo Groeneveld et al. (2024) tokenizer, which performed the best out of the recent tokenizers evaluated. Tokenizers from the BERT and RoBERTa generation of encoder models had competitive downstream performance on MNLI, but we theorized that their lack of recent training data and lack of code support would hinder downstream applications. Interestingly, we observed significant downstream performance degradation when using the Llama 2 Touvron et al. (2023) tokenizer.

## Appendix E Extended results

### E.1 Full GLUE results

The results for all the models each GLUE subsets are presented in Table 5 . The values for prior models are extracted from the literature. As mentioned in Section 3.1.1 , we follow standard practice Liu et al. (2019a) ; Portes et al. (2023) ; He et al. (2023) and conduct an hyperparameter search on each subset. More specifically, we perform a sweep over learning rates in [ 1 ​ e − 5 , 3 ​ e − 5 , 5 ​ e − 5 , 8 ​ e − 5 ] [1e{-5},3e{-5},5e{-5},8e{-5}] , weight decay in [ 1 ​ e − 6 , 5 ​ e − 6 , 8 ​ e − 6 , 1 ​ e − 5 ] [1e{-6},5e{-6},8e{-6},1e{-5}] , and number of epochs in [ 1 , 2 , 3 ] [1,2,3] for tasks in SST-2, MNLI, and RTE, and [ 2 , 5 , 10 ] [2,5,10] for tasks in QNLI, QQP, CoLA, MRPC, and STS-B. The final values are detailed in Table 6 . Early stopping is used for all the fine-tuning runs which reduces the overall fine-tuning time considerably. RTE MRPC and STS-B checkpoints are trained starting from the MNLI checkpoint.

### E.2 Full BEIR results