Smarter, Better, Faster, Longer: A Modern Bidirectional Encoder for Fast, Memory Efficient, and Long Context Finetuning and Inference — part 1/4

Cleaned-text code points [0, 15971). Bibliography omitted here; source.md includes it.

# Smarter, Better, Faster, Longer : A Modern Bidirectional Encoder for Fast, Memory Efficient, and Long Context Finetuning and Inference

###### Abstract

Encoder-only transformer models such as BERT offer a great performance-size tradeoff for retrieval and classification tasks with respect to larger decoder-only models. Despite being the workhorse of numerous production pipelines, there have been limited Pareto improvements to BERT since its release. In this paper, we introduce ModernBERT, bringing modern model optimizations to encoder-only models and representing a major Pareto improvement over older encoders. Trained on 2 trillion tokens with a native 8192 sequence length, ModernBERT models exhibit state-of-the-art results on a large pool of evaluations encompassing diverse classification tasks and both single and multi-vector retrieval on different domains (including code). In addition to strong downstream performance, ModernBERT is also the most speed and memory efficient encoder and is designed for inference on common GPUs.

## 1 Introduction

After the release of BERT Devlin et al. (2019) , encoder-only transformer-based Vaswani et al. (2017) language models dominated most applications of modern Natural Language Processing (NLP). Despite the rising popularity of Large Language Models (LLMs) such as GPT Radford et al. (2018) ; Radford et al. (2019) ; Brown et al. (2020) , Llama Touvron et al. (2023) ; Dubey et al. (2024) , and Qwen Bai et al. (2023) ; Yang et al. (2024) , encoder-only models remain widely used in a variety of non-generative downstream applications. † † https://github.com/AnswerDotAI/ModernBERT

The encoder’s popularity is largely due to their modest inference requirements, enabling them to efficiently process corpora of documents at scale for retrieval and quickly perform discriminative tasks. Encoder models offer a compelling trade-off in quality versus size, making them a popular option against encoder-decoder and decoder-only language models when dealing with substantial amounts of data Penedo et al. (2024) .

Encoder models are particularly popular in Information Retrieval (IR) applications, e.g., semantic search, with notable progress on leveraging encoders for this task Karpukhin et al. (2020) ; Khattab and Zaharia (2020) . While LLMs have taken the spotlight in recent years, they have also motivated a renewed interest in encoder-only models for IR. Indeed, encoder-based semantic search is a core component of Retrieval-Augmented Generation (RAG) pipelines Lewis et al. (2020) , where encoder models are used to retrieve and feed LLMs with context relevant to user queries.

Encoder-only models are also still frequently used for a variety of discriminative tasks such as classification Tunstall et al. (2022) or Natural Entity Recognition (NER) Zaratiana et al. (2024) , where they often match the performance of specialized LLMs. Here again, they can be used in conjunction with LLMs, for example detecting toxic prompts Ji et al. (2023) ; Jiang et al. (2024b) and preventing responses, or routing queries in an agentic framework Yao et al. (2023) ; Schick et al. (2023) .

Surprisingly, these pipelines currently rely on older models, and quite often on the original BERT itself as their backbone Wang et al. (2022) ; Xiao et al. (2023) , without leveraging improvements developed in recent years. Practitioners face many drawbacks: sequence lengths limited to 512 tokens, suboptimal model design Anthony et al. (2024) and vocabulary sizes Karpathy (2023) , and generally inefficient architectures, whether in terms of downstream performance or computational efficiency. Finally, training data is limited in volume and restricted to narrow domains (especially lacking code data) or lacking knowledge of recent events.

Recent modernization efforts have only partially addressed the shortcomings of encoder-only models due to limited breadth. MosaicBERT Portes et al. (2023) , CrammingBERT Geiping and Goldstein (2023) , and AcademicBERT Izsak et al. (2021) focused on matching BERT performance with better training efficiency. NomicBERT Nussbaum et al. (2024) and GTE-en-MLM Zhang et al. (2024) (developed concurrently to this work) introduced longer-context encoder models focused on retrieval applications, but did not optimize for efficiency or classification performance, and re-used older training data mixtures which is especially apparent in programming-related tasks.

Contributions We present ModernBERT, a modernized encoder-only transformer model, with an improved architecture designed to increase downstream performance and efficiency, especially over longer sequence lengths. We also bring encoder-only models to modern, larger data scales, by training on 2 trillion tokens, with a data mixture including code data. We release two models, ModernBERT-base and ModernBERT-large , which reach state-of-the-art overall performance against all existing encoder models on a wide variety of downstream tasks. These results are achieved with considerably higher inference efficiency, processing sequences of 8192 tokens almost two times faster than previous models.

To support future research on encoder-only models, we release FlexBERT 1 1 1 FlexBERT is built on top of a revised MosaicBERT Portes et al. (2023) codebase. , our modular architecture framework allowing easy experimentation, and inspired by Pythia Biderman et al. (2023) , all intermediate training checkpoints (further detailed in Section 2.2.2 ).

## 2 Methods

### 2.1 Architectural Improvements

Our model architecture extends the standard transformer architecture Vaswani et al. (2017) by incorporating extensively tested recent advances (Section 2.1.1 ). We introduce additional efficiency-oriented modifications, through both architectural and implementation improvements (Section 2.1.2 ) and a GPU optimized model design (Section 2.1.3 ). All of our architectural decisions were informed by ablations, which we detail in Appendix D .

#### 2.1.1 Modern Transformer

Bias Terms Following Dayma et al. (2021) , we disable bias terms in all linear layers except for the final decoder linear layer 2 2 2 While many efficient BERT training recipes disable the bias term in the decoder, e.g. Geiping and Goldstein (2023) , we hypothesized a decoder bias might help alleviate weight tying’s negative effects Gao et al. (2019) ; Welch et al. (2020) . . We also disable all bias terms in Layer Norms Xu et al. (2019) . These two changes allow us to spend more of our parameter budget in linear layers.

Positional Embeddings We use rotary positional embeddings (RoPE) Su et al. (2024) instead of absolute positional embeddings. This choice is motivated by the proven performance of RoPE in short- and long-context language models Black et al. (2022) ; Dubey et al. (2024) ; Gemma et al. (2024) , efficient implementations in most frameworks, and ease of context extension.

Normalization We use a pre-normalization block Xiong et al. (2020) with the standard layer normalization Lei Ba et al. (2016) , which is known to help stabilize training Xiong et al. (2020) . Similar to CrammingBERT Geiping and Goldstein (2023) which also uses pre-normalization, we add a LayerNorm after the embedding layer. To avoid repetition, we remove the first LayerNorm in the first attention layer.

Activation We adopt GeGLU Shazeer (2020) , a Gated-Linear Units (GLU)-based Dauphin et al. (2017) activation function built on top of the original BERT’s GeLU Hendrycks and Gimpel (2016) activation function. This is in line with recent work showing consistent empirical improvements when using GLU variants Shazeer (2020) ; Geiping and Goldstein (2023) .

#### 2.1.2 Efficiency Improvements

Alternating Attention Following recent work on efficient long context models Gemma et al. (2024) , attention layers in ModernBERT alternate between global attention, where every token within a sequence attends to every other token, and local attention, where tokens only attend to each other within a small sliding window Beltagy et al. (2020) . In ModernBERT, every third layer employs global attention with a RoPE theta of 160,000 and the remaining layers use a 128 token, local sliding window attention with a RoPE theta of 10,000.

Unpadding ModernBERT follows MosaicBERT Portes et al. (2023) and GTE Zhang et al. (2024) in employing unpadding Zeng et al. (2022) for both training and inference. Encoder-only language models typically use padding tokens to ensure a uniform sequence length in a batch, wasting compute on semantically empty tokens. Unpadding avoids this inefficiency by removing padding tokens, concatenating all sequences from a minibatch into a single sequence, and processing it as a batch of one. Prior unpadding implementations unpad and repad sequences internally for different model layers, wasting compute and memory bandwidth. We use Flash Attention’s variable length attention and RoPE implementations, allowing jagged attention masks and RoPE applications on one unpadded sequence. ModernBERT unpads inputs before the token embedding layer and optionally repads model outputs leading to a 10-to-20 percent performance improvement over other unpadding methods.

Flash Attention Flash Attention Dao et al. (2022) is a core component of modern transformer-based models, providing memory and compute efficient attention kernels. At the start of this work, Flash Attention 3 Shah et al. (2024) , the most recent iteration for Nvidia H100 GPUs, did not include support for sliding window attention. ModernBERT uses a mixture of Flash Attention 3 for global attention layers and Flash Attention 2 Dao (2023) for local attention layers.

torch.compile We leverage PyTorch’s built-in compiling Ansel et al. (2024) to improve the training efficiency by compiling all compatible modules. This yields a 10 percent improvement in throughput with negligible compilation overhead.

#### 2.1.3 Model Design

At the same parameter count, models with more narrow layers ( Deep & Narrow ) have different learning patterns than models with fewer wide layers ( Shallow & Wide ) Nguyen et al. (2021) . Tay et al. (2022) and Liu et al. (2024) have shown that Deep & Narrow language models have better downstream performance than their shallower counterparts, at the expense of slower inference.

Anthony et al. (2024) highlighted that large runtime gains can be unlocked by designing models in a hardware-aware way, which had previously been anecdotally observed by many practitioners Shoeybi et al. (2019) ; Karpathy (2023) ; Black et al. (2022) . ModernBERT was designed through many small-scale ablations to maximize the utilization of a basket of common GPUs 3 3 3 Which, at the time of this work, are server GPUs: NVIDIA T4, A10, L4, A100, and H100 and consumer GPUs: NVIDIA RTX 3090 and 4090. Prioritization was given to inference GPUs (excluding A100 & H100). , while aiming to be as Deep & Narrow as possible without a significant inference slowdown.

ModernBERT has 22 and 28 layers for the base and large models, for a total parameter count of 149 and 395 million, respectively, striking the balance between downstream performance and hardware efficiency. ModernBERT base has a hidden size of 768 with a GLU expansion of 2,304, while large has a hidden size of 1,024 and GLU expansion of 5,248. These ratios allow optimal tiling across tensor cores and the most efficient tiling across the differing number of streaming multiprocessors on our target basket of GPUs. More details on model design are provided in Appendix B .

### 2.2 Training

#### 2.2.1 Data

Mixture Both ModernBERT models are trained on 2 trillion tokens of primarily English data from a variety of data sources, including web documents, code, and scientific literature, following common modern data mixtures. We choose the final data mixture based on a series of ablations.

Tokenizer Unlike the majority of recent encoders which reuse the original BERT tokenizer Nussbaum et al. (2024) ; Portes et al. (2023) ; Zhang et al. (2024) , we opt to use a modern BPE tokenizer. We use a modified version of the OLMo tokenizer Groeneveld et al. (2024) which provides better token efficiency and performance on code-related tasks. The ModernBERT tokenizer uses the same special tokens (e.g., [CLS] and [SEP] ) and templating as the original BERT model Devlin et al. (2019) , facilitating backwards compatibility. To ensure optimal GPU utilization Anthony et al. (2024) ; Karpathy (2023) , the vocabulary is set to 50,368, a multiple of 64 and includes 83 unused tokens to support downstream applications.

Sequence Packing In order to avoid high minibatch-size variance within our training batches as a result of unpadding, we adopt sequence packing Raffel et al. (2020) ; Krell et al. (2022) with a greedy algorithm, which resulted in a sequence packing efficiency of over 99 percent, ensuring batch size uniformity.

#### 2.2.2 Training Settings

MLM We follow the Masked Language Modeling (MLM) setup used by MosaicBERT Portes et al. (2023) . We remove the Next-Sentence Prediction objective which introduces noticeable overhead for no performance improvement Liu et al. (2019a) ; Izsak et al. (2021) , and use a masking rate of 30 percent, as the original rate of 15 percent has since been shown to be sub-optimal Wettig et al. (2023) .

Optimizer We use the StableAdamW optimizer Wortsman et al. (2023) , which improves upon AdamW Loshchilov and Hutter (2019) by adding Adafactor-style Shazeer and Stern (2018) update clipping as a per-parameter learning rate adjustment. StableAdamW’s learning rate clipping outperformed standard gradient clipping on downstream tasks and led to more stable training. Hyperparameters details are given in Appendix A .

Learning Rate Schedule During pretraining, we use a modified trapezoidal Learning Rate (LR) schedule Xing et al. (2018) , also known as Warmup-Stable-Decay (WSD) Zhai et al. (2022) ; Hu et al. (2024) . After a short LR warmup, the trapezoidal schedule holds the LR constant for the majority of training, followed by a short LR decay. This schedule has been shown to match the performance of cosine scheduling Hägele et al. (2024) ; Hallström et al. (2024) with the benefit of enabling continual training on any checkpoint without cold restart issues Ash and Adams (2019) . Unlike most trapezoidal schedules, we use a 1 − sqrt 1-\text{sqrt} LR decay Hägele et al. (2024) , as we found it to outperform linear and cosine decay.

We trained ModernBERT-base at a constant LR of 8e-4 for 1.7 trillion tokens following a 3 billion token warmup. After a 2 billion token warmup, we trained ModernBERT-large at a LR of 5e-4 for 900 billion tokens. We rolled back and restarted training at 5e-5 for the remaining 800 billion tokens after large’s loss plateaued for a few hundred billion tokens at 5e-4.

Batch Size Schedule Batch size scheduling starts with smaller gradient accumulated batches, increasing over time to the full batch size. In ablations, this schedule accelerated training progress. We warmup the batch size from 768 to 4,608 over 50 billion tokens and from 448 to 4,928 over 10 billion tokens, for ModernBERT-base and -large, respectively, with an uneven token schedule so each batch size has the same number of update steps. Details are provided in Appendix A.1 .

Weight Initialization and Tiling We initialize ModernBERT-base with random weights following the Megatron initialization Shoeybi et al. (2019) . For ModernBERT-large, we follow the Phi model family Li et al. (2023) ; Javaheripi et al. (2023) 4 4 4 As detailed in their 2023 NeurIPS presentation. and initialize -large’s weights from ModernBERT-base. In ablation runs, this consistently matched Phi’s improved training results and greatly speed up the initial loss decrease of our model training 5 5 5 This initialization reduced the amount of batch size and LR warmup needed for ModernBERT-large . Details are provided in Appendix A.2 .