Cramming: Training a Language Model on a Single GPU in One Day — part 3/3

Cleaned-text code points [31071, 43663). Bibliography omitted here; source.md includes it.

Of these four sources, we find the Pile to perform best in terms of downstream MNLI performance. However, it turns out we can further improve especially the C4 datset through additional processing. We first evaluate deduplication as described in Lee et al. (2022) via exact substring deduplication, but find this not to help in downstream performance in our case. We then test filtering for uncompressible data. We use the tokenizer itself to remove all training sequences from C4 set that cannot be compressed well; we simply set a threshold t t , e.g. t = 0.3 t=0.3 , and drop all entries from the dataset where the number of tokens in the entry is larger than t t times the number of raw characters. This removes, for example, sequences consisting of hard-to-compress HTML or markdown code. Surprisingly, this results in a measurable improvement on C4, summarized in Table 2 .

We then see some further improvements from two directions. First, sorting all tokenized sequences by some metric, and second, increasing the final batch size. For filtering we sort all tokenized sequences by their average (unigram) token prevalence, so that likely sequences occur first. This has some positive effect, and can be strengthened slightly by drawing from a larger corpus, as the unlikely sequences never get reached. Finally, increasing the batch size to 4032 / 4096 4032/4096 at the end of training (as mentioned in Section 4.3 ) is disproportionally effective on C4, but less so on bookcorpus-wikipedia . We believe that both modifications ultimately reduce the likelihood of training being hindered by fluctuations in the data distribution.

##### Vocabulary Size

We also check whether the original vocabulary size of 32768 32768 described in ( Devlin et al., 2019 ) is optimal in the crammed regime. A priori, this might not hold: The larger, the vocabulary, the more, unique tokens and relationships between unique tokens have to be learned during training. On the other hand, increasing the vocabulary size would compress data further (albeit vanishingly after some point), which would allow for more information to be compressed into the fixed number of tokens that can be ingested during the crammed training run. In Figure 3 , we find that for bookcorpus-wikipedia data, larger vocabulary sizes correlate with larger average GLUE score, although the effect is plateauing for the MNLI task around the original 32768 32768 vocabulary size. Moving forward, we hence keep this vocabulary size.

## 5 Finetuning Performance on GLUE

Finally, we systematically evaluate performance on the GLUE benchmark of Wang et al. (2018) , minus WNLI as in Devlin et al. (2019) . We note that we only use MNLI (m) during the previous sections and do not tune hyperparameters based on the full GLUE scores. We finetune both the pretrained BERT-base checkpoint and our models under the same constraints laid out in Section 2 . For BERT-base, we finetune all datasets for 5 5 epochs with a batch size of 32 32 and learning rate of 2 × 10 − 5 2\text{\times}{10}^{-5} . For the crammed models, we find that this is not optimal and minor improvements can be gained from a batch size of 16 16 and learning rate of 4 × 10 − 5 4\text{\times}{10}^{-5} with cosine decay (this setup does not improve the pretrained BERT checkpoint).

Table 3 and Table 4 describe the performance of this setup on the GLUE downstream tasks (as median over 5 downstream trials). There we compare the original BERT-base checkpoint, a reproduction of the BERT pretraining settings stopped after our budget is reached, the setup described in ( Izsak et al., 2021 ) and the modified recipe, trained for a single day for each GPU setup. Overall, performance is surprisingly decent, especially for the larger datasets of MNLI, QQP, QNLI and SST-2, where downstream finetuning can smooth out the remaining differences between the full BERT model and the crammed variants. Further, we find substantial gains over both a naive BERT training with limited budget, and over the recipe described in ( Izsak et al., 2021 ) . For ( Izsak et al., 2021 ) , the described recipe was originally designed for a full 8 GPU server blade, and squeezing the BERT-large model therein onto the smaller GPUs in this experiment is resposnsible for most of the performance degradation of this recipe in our scenario.

Overall, the crammed model mostly works, even for smaller datasets. The average is brought down however by a significant drop on CoLA (corpus of linguistic acceptability) ( Warstadt et al., 2019 ) .

This behavior is intriguing and we offer two hypotheses. First, it is conceivable that the chosen global hyperparameters for finetuning are a bad fit for CoLA in particular. CoLa performance can be brittle with respect to hyperparameter, with Jiao et al. (2020) training longer only on CoLA or Joshi et al. (2020) training less only on CoLA. Nevertheless, for BERT, a set of global hyperparameters exists, pointing at a deficiency in the crammed model. As a second hypothesis, it is conceivable that these models need to process more text before they memorize enough data to do well on CoLA. This would be in contrast to Liu et al. (2021d) , who find that CoLA is learned relatively quickly compared to other downstream tasks when probing intermediate BERT checkpoints. On the other hand, deficiencies on CoLA in particular are also common in approaches that distill BERT into smaller architectures ( Sun et al., 2019 ; Turc et al., 2019 ; Mukherjee et al., 2021 ) , which might come with limited capacity for linguistic acceptability.

### 5.1 Ablation - Which Changes Really Mattered?

In Table 5 we provide a summary ablation study of all changes discussed in this work. We group modifications, as in previous sections into the three groups of architecture, training and data and ablate each group by resetting all modifications to the original BERT recipe. Here, we find that we first have to make minimal modifications in any case, as modifications to architecture, such as PreNorm layer structures also in turn allow the more aggressive learning rate schedules described in the training setup. Taking this into account, we ultimately find about two percentage points gained in average GLUE score through architectural changes, one percentage point in data changes, and half a precentage point in training modifications.

### 5.2 What Happens When Training Longer?

We also verify what happens if the cramming recipe discussed so far is used with more budget. To this end, we train models for 48 hours on 8 A6000 GPUs, which ends up to be 208 208 total exaFLOP, c.f. Table 1 . We directly apply the setting described so far, simply scaling the learning rate schedules to cover the new budget of 48 hours. In Table 6 , we find that the discussed recipe does immediately generalize to larger compute budgets. This is surprising, not the least, as now, the dataset (which was sorted in Section 4.4 is now too small and repeated multiple times. The newly trained models have strong performances, especially on MNLI and SST-2, where they significantly outperform the original BERT checkpoint and fall into a similar range as the roBERTA-base checkpoint of Liu et al. (2019) , which was trained with much more compute. Yet, in other tasks, such as (again) CoLA, the new models barely improve even in the larger compute regime.

## 6 Limitations

In this work, we limited our investigation to transformer-based architectures trained with MLM objectives. However, we do think that the general task of cramming posed in Section 2 is interesting even when relaxing these constraints. There have been a number of modifications proposed to the objective in particular ( Joshi et al., 2020 ; Bao et al., 2020 ; Bajaj et al., 2022 ; Tay et al., 2022b ) . While Artetxe et al. (2022) and Wang et al. (2022) find MLM still to hold up well as a pretraining objective, other suggestions such as ELECTRA ( Clark et al., 2019 ; Clark et al., 2020 ; He et al., 2021 ) could be employed which might be beneficial for crammed models. Also, the optimal architecture might not be transformer-based ( Merity, 2019 ; Fusco et al., 2022 ; Peng, 2021 ) .

## 7 Conclusions

We discuss how much performance a transformer-based language model can achieve when crammed into a setting with very limited compute, finding that several strands of modification lead to decent downstream performance on GLUE. Overall though, cramming language models appears hard, as we empirically find many implications of Kaplan et al. (2020) to still hold in this regime, and for examples improvements through larger models are evened out by their slower speed. We hope that this work can provide a baseline for explorations of the question of cramming we formalize in Section 2 and cast an additional light on a number of improvements and tricks proposed for transformer architectures in recent years.

#### Reproducibility Statement

We provide code to reproduce all experiments at https://github.com/JonasGeiping/cramming .

## Appendix

## Appendix A Other Modifications

A few recent developments not included in this study are Roy et al. (2022) , Shen et al. (2022) , and Mindermann et al. (2022) . Modifications further not included in this study are more involved initialization ( Zhu et al., 2021 ) , additional objective modifications ( Müller et al., 2019 ) , progressive growth ( Gu et al., 2021 ; Shen et al., 2022 ) , convolutional variants ( Iandola et al., 2020 ; Chelombiev et al., 2021 ; So et al., 2021 ) , sequence recurrence ( Lei et al., 2022 ) and TUPE embeddings ( Ke et al., 2020 ) .

## Appendix B Additional Information

Additional results concerning architecture modifications can be found in Table 8 and Table 9 . Additional results for training modifications can be found in Table 10 . Not all results remarked on in the main body (especially for variations that did not work, marked in gray) are accompanied by raw results in this appendix, but can be computed using the provided implementation.

### B.1 References for Table 1

The maximal floating point operations referenced in Table 1 are based on the following published numbers. For TPU specs, according to https://cloud.google.com/tpu/docs/system-architecture-tpu-vm we find 275 TFLOP/s in bfloat16 precision for the TPUv4 and 123 TFLOP/s for the TPUv3, each per chip. The V100 peak performance is given as 125 TFLOP/s in https://images.nvidia.com/content/volta-architecture/pdf/volta-architecture-whitepaper.pdf in ”TFLOPS of mixed precision”. Some NVIDIA datasheets also reference TFLOP/s with sparsity, which are not applicable in the context of this work. The Titan RTX comes out at 130.5 TFLOP/s, in ”Peak FP16 Tensor TFLOPS with FP32 Accumulate” as described in https://images.nvidia.com/aem-dam/en-zz/Solutions/geforce/ampere/pdf/NVIDIA-ampere-GA102-GPU-Architecture-Whitepaper-V1.pdf . For the A6000, we find 154.8 ”Peak BF16 Tensor TFLOPS with FP32 Accumulate” also in https://www.nvidia.com/content/PDF/nvidia-ampere-ga-102-gpu-architecture-whitepaper-v2.pdf . A4000 performance is actually less clear from this whitepaper, and estimated to be 88.45 TFLOP/s, based on it containing 192 tensor cores, compared to 336 for the A6000. For the RTX2080ti, the whitepaper at https://images.nvidia.com/aem-dam/en-zz/Solutions/design-visualization/technologies/turing-architecture/NVIDIA-Turing-Architecture-Whitepaper.pdf reports 53.8 ”peak FP16 Tensor TFLOPS with FP32 Accumulate” for the reference edition. All total exaFLOP numbers are then computed based on these TFLOP/s numbers over the training time period described in each work.

## Instructions for reporting errors

We are continuing to improve HTML versions of papers, and your feedback helps enhance accessibility and mobile support. To report errors in the HTML that will help us improve conversion and rendering, choose any of the methods listed below:

Click the "Report Issue" ( ) button, located in the page header.

Tip: You can select the relevant text first, to include it in your report.

Our team has already identified the following issues . We appreciate your time reviewing and reporting rendering errors we may not have found yet. Your efforts will help us improve the HTML versions for all readers, because disability should not be a barrier to accessing research. Thank you for your continued support in championing open access for all.

Have a free development cycle? Help support accessibility at arXiv! Our collaborators at LaTeXML maintain a list of packages that need conversion , and welcome developer contributions .