No Train No Gain: Revisiting Efficient Training Algorithms For Transformer-based Language Models — part 2/3

Cleaned-text code points [15550, 31387). Bibliography omitted here; source.md includes it.

We also compare the performance of dynamic architecture methods methods on the T5-base model evaluated on the SNI benchmark and report the results in Table 2 . Here layer stacking significantly improves over the baseline for 6 and 12 hour budgets. However, given 24 hours for training, the final accuracy of layer stacking reduces back to the accuracy of the 6 hour model. Here the baseline significantly improves over all methods. Notably, for all budgets, layer dropping is consistently outperformed by the baseline .

## 5 Case Study 2: Batch Selection

Scaling up the size of web-crawled data for pre-training has been one of the major drivers to improve the capabilities of language models [ 43 , 10 ] . A line of work has argued that training speed-ups emerge through the selection of informative examples. They argue that these examples can be identified from certain statistics collected throughout training [ 4 , 44 , 38 , 45 , 71 ] . Here, we focus on two such methods that directly alter the training procedure to steer gradient computations towards informative samples by subsampling examples from a mega-batch to curate the mini-batch, called batch selection .

### 5.1 Selective Backprop

Due to its simplicity, we first examine selective backprop [ 38 ] , described in Algorithm 3 . The high-level idea of selective backprop is to compute the backward pass only on the training examples with the highest loss. To construct such batches, it first computes the loss of every example in a uniformly-sampled mega-batch. It then samples a high-loss subset of the mega-batch by ranking points based on their loss percentiles w.r.t. historical losses among recently-seen inputs.

### 5.2 RHO Loss

Mindermann et al. [65] argue that solely prioritizing high training loss results in two types of examples that are unwanted: (i) mislabeled and ambiguous data, as commonly found in noisy, web-crawled data; and (ii) outliers, which are less likely to appear at test time. The authors propose down-weighting such data via a selection objective called Reducible Holdout (RHO) loss , shown in Algorithm 4 .

The authors acknowledge that their method comes with three overhead costs: (1) pre-training a proxy model using holdout data, (2) one extra forward pass over the entire training set to store the proxy model’s training losses, and (3) additional forward passes for the batch selection. In our evaluations we never count the costs of (1) and (2) because, as Mindermann et al. [65] point out, these costs can be amortized over many training runs. For (1), we pre-train a model for 6 hours of RST on held-out data, which is enough to reach reasonable performances ( Table 1 ).

Despite Mindermann et al. [65] motivating their method for a scenario where additional workers are available to perform batch selection, we wondered if it might still provide performance gains even if this is not the case. Mindermann et al. [65] do not implement or evaluate an algorithm where extra workers are used to select batches. Because of this, we evaluate RHO loss under two protocols: in the main results, we count the batch selection costs (3) against the training budget, while in Section 5.4 , we provide a second set of results where we ignore the cost of batch selection.

### 5.3 Results

We assume that the effects of selecting better training batches should be largely agnostic to whether we pre-train a BERT or T5 model. Hence, instead of training both architectures, we decide to pre-train only BERT models and instead vary the datasets and budgets as follows.

For the first set of experiments, we fix the budget to 12 hours and consider three different pre-training datasets: (i) C4 [ 76 ] , consisting only of webpage text which, despite being regularly used for pre-training, is known to have quality issues [ 52 ] , (ii) Bookcorpus and Wikipedia [ 22 ] , which contain polished, book(-like) text and MiniPile [ 40 ] , a subset of the diverse Pile pre-training corpus [ 28 ] , containing code, mathematics, books, webpages, and other scientific articles.

#### Validation loss.

To rank the pre-training performances, we compare the validation loss , which is directly comparable as we use the same inputs for all methods (which is not the case for the training data). This is shown in Figure 3 , which shows the validation loss every 3 hours throughout training. We find that both batch selection methods do not improve over the baseline.

#### Downstream performance.

Next, we investigate downstream performances, fix the C4 corpus as the pre-training corpus, and vary the budgets (6, 12, and 24 hours). Figures 16 and 2 show the results on SuperGLUE and GLUE. We observe very small differences between the methods and the baseline . On average (Table 1 ) no batch selection method significantly outperforms the baseline .

### 5.4 Ablation: Batch Selection For Free

We want to disentangle whether batch selection methods fail to improve over the baseline because their gains do not compensate for their computational overhead. In the previous section ( Section 5.3 ) and the main results ( Table 1 ), we accounted for this overhead. Therefore, in those experiments, selective backprop and RHO loss effectively update the model for fewer iterations than the baseline within the fixed budgets. Here, we re-run a small number of experiments where batch selection is free, and so we train these models with the same number of iterations as the baseline.

Specifically, we run an experiment for 12 hours using the C4 corpus and GLUE downstream tasks. For selective backprop , we choose β = 1 \beta=1 , which resulted in ~ 1.7 1.7 x of the wall-clock time; while for RHO loss , we choose a mega-batch size that is 10 10 x larger (15360) than the mini-batch size (1536), following Mindermann et al. [65] , which led to ~ 5.3 5.3 x of the original pre-training time. Table 4 shows the results. We see that RHO loss reaches a slightly better final validation loss but performs worse on the GLUE downstream tasks than baseline and selective backprop , which does not improve over the baseline .

## 6 Case Study 3: Efficient Optimizers

Recently, two efficient optimizers were proposed to replace the ubiquitous Adam(W) optimizers [ 47 , 61 ] for language models: Lion [ 13 ] and Sophia [ 57 ] .

Lion [ 13 ] is an optimizer symbolically discovered in the vast program space of first-order optimization primitives. As such, it does not follow any theory-grounded principle that would justify its acceleration property a priori; however, Chen et al. [13] report empirical speed-ups over AdamW in many settings.

Sophia [ 57 ] is a scalable stochastic second-order optimizer primarily designed for and evaluated on language model pre-training. The authors claim that Sophia achieves a 2x speed-up compared with AdamW in the number of steps, total compute, and wall-clock time. The authors study two Hessian estimators, but as of this writing, only open-source the code for the empirically better one (Gauss-Newton-Bartlett), which we use here.

The baseline in this section simply refers to AdamW [ 61 ] .

#### Mixed Precision Training with BF16 vs. FP16

A common practice for training language models is to use mixed precision training [ 64 ] . In initial experiments with BERT, we observed several numerical instabilities (NaN training losses) during hyper-parameter search after inserting Lion and Sophia into our training pipelines as drop-in replacements. Our default mixed-precision mode was FP16, and as noted by other practitioners of Sophia [ 56 ] , BF16 can sometimes be more stable. Hence; for the optimizer comparisons with BERT, we report results in BF16; including the baseline (although we notice that the baseline ’s training curves are essentially identical across both modes). For the T5 model, we use the TF32 precision format.

#### RMS-Scaling for T5 Pre-training

T5 pre-training [ 76 , 81 ] typically employs the Adafactor optimizer [ 82 ] , which relies on tensor-wise learning rate scaling determined by the tensor’s root mean square (RMS). This scaling has been identified as critical for convergence during pre-training when using AdamW [ 68 ] . Initial tests with Lion and Sophia without additional adjustments led to divergence for higher learning rates or suboptimal performance. This mirrored the behavior of AdamW without RMS scaling. To address this, we incorporated RMS scaling into Lion and Sophia for a subsequent round of experiments, which we outline in Section C.6 .

### 6.1 Results

In the case of BERT downstream performances ( Table 1 ), we find that Lion and Sophia perform about the same as the baseline . Figure 6 shows the results in more detail. We note that baseline (AdamW) consistently ranks the highest and has a comparatively low standard deviation over random seeds.

Analogous to the batch selection for free ablation in Section 5.4 , we also experiment with Sophia while not counting for Hessian update steps and running for the same number of iterations, as shown in Figure 4 . Surprisingly, we still do not observe any speedup.

## 7 Conclusion

In this work, we closely examined efficient training algorithms which promised to deliver training speed-ups. First, we clarify that quantifying a training speed-up without specifying an explicit training budget can be misleading and that some previous works missed this. To normalize the wall clock time across different hardware, we introduce the reference system time measure. Then, we put three classes of algorithms to the test in various pre-training settings of BERT and T5 models. We found only a few settings where some of the considered algorithms improved over the baseline.

## Acknowledgments

The authors would like to thank Edoardo Ponti for his feedback on an earlier version of this manuscript. JK and OK acknowledge support from the Engineering and Physical Sciences Research Council with grant number EP/S021566/1. OK was supported by G-Research. PN was supported by the UKRI Centre for Doctoral Training in Natural Language Processing, funded by the UKRI (grant EP/S022481/1) and the University of Edinburgh, School of Informatics and School of Philosophy, Psychology & Language Sciences. PM was partially funded by the European Union’s Horizon 2020 research and innovation programme under grant agreement no. 875160, ELIAI (The Edinburgh Laboratory for Integrated Artificial Intelligence) EPSRC (grant no. EP/W002876/1), an industry grant from Cisco, and a donation from Accenture LLP; and is grateful to NVIDIA for the GPU donations. This work was supported by the Edinburgh International Data Facility (EIDF) and the Data-Driven Innovation Programme at the University of Edinburgh.

## Appendix A Limitations and Future Work

Firstly, while we ran extensive ablations within the 6-, 12-, and 24-hour training regimes, it is possible that our results do not generalize to much longer ones. We justify this choice for two reasons. Firstly, all downstream task performances observed are only a few percentage points worse than state-of-the-art performances; e.g., our 24-hour BERT model reaches ~ 79 79 % / 58.5 % 58.5\% on GLUE/SuperGLUE, while fine-tuning the original BERT-base model 3 3 3 https://huggingface.co/bert-base-uncased reaches 80.9 80.9 % / 60.8 60.8 %, respectively. Similarly, our 24h-T5 model reaches 39.5 39.5 % on SNI, while using the original checkpoint, reaches 41.0 41.0 % [ 68 ] . Secondly, we argue that an efficient training method should work well specifically in settings with limited budgets.

Next, as illustrated in Appendix B , there is an abundance of efficient training algorithms, and rigorously evaluating all of them is prohibitively expensive. Hence, one limitation of this work remains that we did not consider other efficiency-promoting algorithms, and we hope to explore more in future work.

Another limitation is our sole focus on language model pre-training. Investigating approaches for (i) efficient fine-tuning of (large) language models or (ii) pre-training on other data-intensive modalities such as images and video remains promising too. We expect that our experimental protocol utilizing RSTs will benefit future works doing so.

## Appendix B Related Work

### B.1 Efficient Training Algorithms

There is an abundance of proposals for efficient training; surveys of these can be found in [ 8 , 84 , 42 ] . We roughly categorize them into architecture-centric , data-centric , optimization-centric , and others .

#### Architecture-centric strategies.

These decide how to avoid forward/backward passes of specific weights in the network. The idea of gradually growing a network to accelerate training, as in layer stacking , dates back to the 90s [ 25 , 54 ] . There are a number of follow-ups to the layer stacking paper [ 30 ] , including automated stacking using elastic supernets [ 55 ] loss- and training-dynamics preserved stacking [ 85 ] and stacking small pre-trained models to train larger models [ 95 ] .

Methods akin to layer dropping include FreezeOut [ 9 ] , LayerDrop [ 26 ] (which focuses on network sparsity), and AutoFreeze [ 59 ] . In these methods, forward/backward passes for specific layers are skipped based on either a pre-determined schedule [ 9 , 26 ] or based on statistics from prior forward/backward passes [ 59 ] .

While not the focus of this work, a similar motivation can also be found in dynamic sparse training methods, which aim to identify relevant sub-networks during training and promote the prunability of the network [ 66 , 21 , 24 ] . However, these approaches do typically not lead to training speedups since unstructured sparsity is not GPU-friendly, which is why we do not consider them here [ 58 ] .

#### Data-centric strategies.

Besides batch selection methods, which subsample relevant data from a mega-batch throughout training as discussed in Section 5 , there are a number of other approaches aiming to either order or subsample training data. We did not consider them in our experiments since they either have a different motivation than training efficiency or are not directly suitable for a drop-in modification of the training procedure. However, we briefly summarize three classes of such work. Firstly, one of the oldest classes of data-selection strategies for training is curriculum learning (for a recent survey, see Wang et al. [96] ). We do not consider curriculum learning here as it has already been extensively evaluated [ 99 ] , and because it was initially motivated to improve generalization rather than achieve efficiency gains. Secondly, task-specific retrieval [ 32 , 104 ] use task-specific data to retrieve similar data from a large unsupervised corpus. Different from the task-agnostic batch selection methods we consider here, these methods are specifically designed to improve downstream task performance. Lastly, another line of work tries to subsample the entire dataset decoupled from the training procedure by various scoring heuristics [ 16 , 31 , 73 , 53 , 87 , 1 , 101 , 100 , 60 ] , and we believe that further investigating such can be a promising direction for future work.

#### Optimization-centric strategies.

Lots of optimizers have been proposed with the goal of speeding up convergence [ 23 , 107 , 112 ] ; yet, Schmidt et al. [79] report that none of them consistently outperforms Adam(W) [ 47 , 61 ] when put to a rigorous test. Instead of modifying the training procedure, another line of work observed intermediate performance speedups by averaging weights along the training trajectory post-hoc training [ 39 ] ; however, such gains arise primarily through effectively lowering the learning rates without directly intervening in the training process [ 77 , 78 ] , which is different to the in this work considered methods which do intervene.

#### Others.