No Train No Gain: Revisiting Efficient Training Algorithms For Transformer-based Language Models — part 1/3

Cleaned-text code points [0, 15550). Bibliography omitted here; source.md includes it.

# No Train No Gain: Revisiting Efficient Training Algorithms For Transformer-based Language Models

###### Abstract

The computation necessary for training Transformer-based language models has skyrocketed in recent years. This trend has motivated research on efficient training algorithms designed to improve training, validation, and downstream performance faster than standard training. In this work, we revisit three categories of such algorithms: dynamic architectures ( layer stacking , layer dropping ), batch selection ( selective backprop , RHO loss ), and efficient optimizers ( Lion , Sophia ). When pre-training BERT and T5 with a fixed computation budget using such methods, we find that their training, validation, and downstream gains vanish compared to a baseline with a fully-decayed learning rate. We define an evaluation protocol that enables computation to be done on arbitrary machines by mapping all computation time to a reference machine, which we call reference system time . We discuss the limitations of our proposed protocol and release our code to encourage rigorous research in efficient training procedures: https://github.com/JeanKaddour/NoTrainNoGain .

## 1 Introduction

Language models are advancing rapidly, surpassing human-level performances in some experimental setups [ 11 , 46 , 98 , 19 , 12 ] . These improvements are primarily due to model, data, and training budget scaling [ 43 , 34 , 3 ] . Training a single state-of-the-art language model requires hundreds of thousands of GPU hours, costs millions of dollars, and consumes as much energy as multiple average US family households per year [ 80 , 72 , 14 , 91 , 42 ] .

To remedy this, there is a rapidly growing line of work on efficient training algorithms, which modify the training procedure to save computation [ 80 , 8 , 111 , 84 , 63 , 42 ] . Specifically, three distinct classes of such algorithms are dynamic architectures ( layer stacking [ 30 ] and layer dropping [ 106 ] ) which ignore some of the weights during training, batch selection ( selective backprop [ 38 ] and RHO loss [ 65 ] ) which skip irrelevant data, and efficient optimizers ( Lion [ 13 ] and Sophia [ 57 ] ) which claim to convergence faster than Adam(W) [ 47 , 61 ] .

However, we find that the evaluation methodology is not standardized, and there are inconsistencies in the quantification of a “speed-up”. Two general trends are: comparisons of (1) intermediate performances throughout training instead of final ones within a pre-defined training budget, and (2) incomplete speedup metrics , e.g., training progress as a function of the number of iterations (or epochs) instead of wall-clock computation times despite unequal per-iteration costs .

As pointed out by Dehghani et al. [20] , Dahl et al. [17] , this can be unfair to baselines, which were not tuned for efficiency within the same compute budget.

An example for (1) includes learning rate schedules, which can be arbitrarily stretched to improve the final performance while “sacrificing” the quality of intermediate checkpoints [ 102 , 48 , 86 ] . This can make comparisons of intermediate performances unfair to baselines. For (2), additional regularization techniques can improve the per-iteration convergence at higher per-iteration costs [ 5 , 27 , 41 , 17 ] , rendering a wall-clock time comparison more appropriate.

In this paper, we propose a simple evaluation protocol for comparing speedups of efficient training algorithms. We use this protocol to evaluate these algorithms for pre-training Transformer-based language models from scratch. We compare different training budgets (6, 12, and 24 hours), model architectures (BERT-Base [ 22 ] and T5-Base [ 76 ] ), and (for batch selection algorithms) datasets (C4 [ 76 ] , Wikipedia and BookCorpus [ 110 ] , and MiniPile [ 40 ] ). To account for variability in measured wall-clock time on different hardware and software configurations, we propose a simple measure that we call reference system time (RST).

Our key findings are as follows:

## 2 Comparing Efficient Training Algorithms

What is the fairest way to compare efficient training algorithms? Ideally, each method should be given an identical compute budget, and methods that converge to better solutions than a baseline within this budget achieve a speed-up. Below, we discuss the issues of commonly used metrics to specify the compute budget, and propose an improved metric.

#### The Pitfalls of Iterations

Specifying a budget in number of training iterations is usually not suitable because the iteration time can vary between methods.

#### The Pitfalls of FLOPs

While FLOPs count the number of elementary arithmetic operations, they do not account for parallelizability (e.g., RNNs vs Transformers) or hardware-related details that can affect runtime. For example, FLOP counting ignores memory access times (e.g., due to different layouts of the data in memory) and communication overheads, among other things [ 20 , 8 ] .

#### The Pitfalls of Wall-Clock Time

WCT can fluctuate even on the same hardware, for instance, due to the usage of non-deterministic operations 1 1 1 https://pytorch.org/docs/stable/notes/randomness.html , hidden background processes, or inconsequential configurations, such as the clock rate. Further, we would like a metric that allows researchers to run on shared compute clusters, where hardware configurations will vary.

#### Our Proposal: Reference System Time (RST)

We propose a simple time measure that will allow us to standardize any timing result w.r.t. a reference hardware system (e.g., NVIDIA RTX 3090, CUDA runtime 11.8, PyTorch 2.0, etc.). We define the time elapsed on a reference system as reference system time (RST). To convert the training run of an arbitrary device to RST, we first record the time per training iteration on the reference training system. 2 2 2 We average the time required per iteration across 1000 iterations given some model architecture, batch, sequence length, system configuration, etc. Then, we compute the RST by multiplying the number of iterations run on the arbitrary device by the time per iteration on the reference system. This way, the time is grounded in practical time units but can be applied to any system.

### 2.1 The Pitfall of Comparing Intermediate Performances

One difficulty with comparing efficient training methods stems from the importance of the learning rate schedule to model performance. Various works have demonstrated that deep learning models tend to reach their maximum performance only once the learning rate has been fully decayed [ 108 , 75 ] . If the learning rate schedule is stretched, this can delay the number of iterations required for the model to reach a given performance threshold since the optimizer keeps “exploring” the local loss basin, wasting iterations oscillating [ 33 , 41 ] .

This leads to unfair comparisons when the learning rate schedule is not set fairly between methods; a non-obvious pitfall that has been discussed previously in the literature [ 49 , 20 ] , but which we still observed in several of the works we evaluated [ 30 , 106 , 38 , 57 ] . Consider comparing a baseline B with an efficient training method X, where both methods are trained for the same number of iterations, and the learning rate is decayed in terms of the number of iterations. B and X reach a similar performance at the end of training, but because X completes the iterations in less WCT, a speed-up is declared.

However, this is not a fair comparison because the training budget was measured in iterations, but the performance of the models was evaluated at a point specified in WCT, specifically the time it took X to complete the budget. Thus, B is evaluated at an intermediate point during training when its learning rate schedule has not been fully decayed. If the learning rate schedule of B had been shortened so it finished at the same WCT as X, the two methods may have reached similar performances in the same WCT time. The same problem arises when the budget is specified using different units to the point of measurement, including the cases where FLOPs are used for the budget.

To avoid this issue, each method should be trained to a fixed budget with the performance measured at the end of this budget. Importantly, the learning rate and other schedules must be decayed as a function of the same unit used to measure the budget.

## 3 Experimental Setup

Given this computation measure, we can now evaluate efficient training algorithms under fixed computation budgets. We conduct experiments using two established and widely used Transformer language models: BERT [ 22 ] and T5 [ 76 ] . We choose a single-GPU setting following recent works [ 29 , 36 , 68 ] to facilitate reproducibility and access for compute-restricted researchers. However, in principle, our protocol can be run on any hardware and straightforwardly used in a distributed setup.

#### BERT: Encoder-only.

We follow the setup and hyperparameters of Geiping & Goldstein [29] with minor modifications. We pre-train a BERT-base-like model with 16 layers instead of 12, which consists of 120M parameters, using a masked language modeling (MLM) objective. We use the AdamW optimizer [ 61 ] with hyperparameters β 1 = 0.9 \beta_{1}=0.9 , β 2 = 0.98 \beta_{2}=0.98 , ϵ = 10 − 12 \epsilon=10^{-12} , and weight decay of 10 − 2 10^{-2} [ 61 ] . Further, we use a one-cycle learning rate schedule [ 86 ] with a peak learning rate of 10 − 3 10^{-3} and gradient clipping of 0.5 0.5 . The batch size is 1536 1536 sequences, and the sequence length is 128 128 . We fine-tune and evaluate the pre-trained BERT models using the GLUE [ 93 ] and SuperGLUE [ 94 ] benchmarks. For fine-tuning on GLUE, we use the hyper-parameters from Geiping & Goldstein [29] . On SuperGLUE, we tune them using the validation accuracy of BoolQ [ 15 ] . We found inconsistencies in the literature on how to aggregate the SuperGLUE scores; here, we average the following scores: for CB, we use the F1 score; for the rest (BoolQ, COPA, RTE, WiC, WSC), we use the accuracy.

#### T5: Encoder-Decoder.

We pre-train a T5v1.1-Base [ 76 , 81 ] model using the original span-corrupting MLM objective and SentencePiece [ 51 ] tokenizer. We follow Nawrot [68] and use the AdamW optimizer [ 61 ] with tensor-wise LR scaling by its root mean square (RMS), base learning rate 0.02 0.02 , no weight decay, cosine schedule with final of 10 − 5 10^{-5} [ 62 ] , gradient clipping of 1.0 1.0 , and 10 4 10^{4} warm up steps. We use a batch size of 144 144 examples, where each example consists of an input of 512 512 tokens and an output of 114 114 tokens. We evaluate our pre-trained T5 models on the Super-Natural-Instructions [ 97 , SNI,] benchmark. To fine-tune the model on SNI, we strictly follow the hyperparameters of [ 68 , 97 ] . For both pre-training and fine-tuning, we use TF32 precision.

#### Dataset.

Unless specified otherwise, we use the C4 [ 76 ] dataset for less than one epoch (i.e., without data repetitions) without sentence curriculum and de-duplication [ 52 , 29 ] . In Section 5 , we additionally use two other datasets.

#### Learning-rate schedule.

We adjust the learning rate schedule based on the elapsed time conditional on a time budget (measured in RST), similar to [ 37 , 29 ] , who measure raw WCT.

#### Hyper-parameter search.

For all considered methods, we tune their hyper-parameters based on the pre-training loss. We list all details about each method’s hyper-parameters, our considered grid search ranges, and the best values we found in Appendix C .

#### Reference System.

We record the RST on two separate systems for BERT and T5. For BERT, we choose a single NVIDIA RTX 3090, CUDA 11.7, PyTorch 1.13. For T5, we choose an NVIDIA A100, CUDA runtime 11.8, PyTorch 2.0.

## 4 Case Study 1: Dynamic Architectures

### 4.1 Layer stacking

Layer stacking [ 30 ] , as summarized in Algorithm 1 , replicates a L L -layer model into a 2 ​ L 2L -layer model by copying its parameters, effectively warm-starting the stacked model with parameters transferred from the smaller model. Thereby, it benefits from faster per-iteration times in the early training phases when using fewer layers. Gong et al. [30] attribute the success of this method to attention distributions of bottom layers being very similar to attention distributions of top layers, indicating that their functionalities are similar.

### 4.2 Layer dropping

Layer dropping exploits the following observation: the layers of a network do not contribute equally to the loss reduction throughout training [ 35 , 6 , 105 ] . It does so by randomly choosing parts of the model to be skipped during each training step. Specifically, it replaces a subset of Transformer blocks with the identity function. As a result, it reduces the overall computational cost of the model for each training step because it skips these layers during the forward and backward passes.

To minimize the impact of layer dropping on the pretraining loss of the model, Zhang & He [106] employ a time and depth schedule that determines the probability of dropping each block. The time schedule begins with a zero probability of dropping each block. Then it increases this probability throughout the training process until it reaches a maximum of ( 1 − α ¯ ) (1-\bar{\alpha}) , where the hyperparameter α ¯ = 0.5 \bar{\alpha}=0.5 as chosen by Zhang & He [106] .

The depth schedule ensures that blocks located earlier in the model are dropped with a lower probability than those located later/deeper. An important hyperparameter of the depth schedule in layer dropping is γ f \gamma_{f} , which controls the rate at which the probability of dropping layers increases. A higher value of γ f \gamma_{f} leads to a quicker increase in the probability. Zhang & He [106] set γ f \gamma_{f} to 100 in their experiments. We refer the reader to the original work by Zhang & He [106] for more details.

### 4.3 Results

#### Training losses.

Figure 1 shows the pre-training losses for the baseline , layer stacking , and layer dropping on BERT and T5 when given a budget of 24 hours in RST. In both settings, layer stacking achieves the lowest loss within this budget. The gap between layer stacking and the baseline closes almost completely as the budget is increased to 24 hours. However, layer dropping is consistently worse than the baseline throughout training. In both models, the gap between layer dropping and the baseline is larger than between the baseline and layer stacking at the end of training. Further, layer dropping introduces additional fluctuations in the loss when training T5, compared to the baseline .

#### Downstream performance.

Figure 2 shows the SuperGLUE downstream performance after using the baseline and all dynamic architecture methods ( layer stacking and layer dropping ) to pre-train BERT. We evaluate all methods using three different RST budgets (to get a sense of the Pareto-frontier of budget and performance [ 74 ] ): 6 hours, 12 hours, and 24 hours. We notice that on specific SuperGLUE datasets, the efficiency methods have little or even detrimental effect on performance (CB, WiC, WSC, BoolQ). Across all datasets, COPA appears to have the largest gains from efficient training. Averaged across all datasets, efficient training methods have little effect. We report the exact numbers in Table 1 (right). On average, across all budgets, neither layer stacking and layer dropping significantly improve over the baseline .