No Train No Gain: Revisiting Efficient Training Algorithms For Transformer-based Language Models — part 3/3

Cleaned-text code points [31387, 43456). Bibliography omitted here; source.md includes it.

These include ways to improve the faster computation of Transformer building blocks [ 90 , 18 , 69 ] , allocate compute conditioned on the input [ 83 , 70 , 2 ] , network initialization [ 7 , 109 ] .

### B.2 Efficient Training Meta-Studies

#### Budget-centric recipes.

A different line of work investigates budget-centric training recipes, for example, for academic settings with multiple GPUs [ 37 , 113 ] , or hard time constraints such as training on a single GPU for one day [ 36 , 29 , 68 ] . Our work adopts some of the recipes proposed by Geiping & Goldstein [29] , Nawrot [68] and aims to complement them by investigate (other) speedup techniques.

#### Empirical Meta-Studies of Training Transformer-based models.

Narang et al. [67] study Transformer modifications across implementations and applications, finding that most do not meaningfully improve performances. Similarly, Tay et al. [89] study the scaling properties of various Transformer-based architectures (some of which are designed for efficiency), concluding that the original Transformer proposed by Vaswani et al. [92] has the best scaling behavior. Dehghani et al. [20] discuss how common cost indicators of machine learning models have different pros and cons and how they can contradict each other. Further, they show how training time can be gamed. Our work is heavily inspired by Dehghani et al. [20] and aims to complement it in two ways: (1) by proposing to normalize WCT across different systems (using RST) and (2) by a thorough experimental study in the case of pre-training Transformer-based language models.

Closest to our work is the concurrent work by Dahl et al. [17] , who exhaustively discuss various pitfalls of benchmarking neural network optimizers. Among other considerations, they propose to benchmark algorithms within a fixed runtime budget, a practice we agree with and use in our work too. We believe our work additionally complements theirs by other methods beyond optimizers.

## Appendix C Hyper-Parameter Search

### C.1 Layer stacking : When To Stack

Figure 7 illustrates that layer stacking has relatively low sensitivity to different stacking RST hour times, namely { ( 2.4 , 7.2 ) , ( 3 , 7.2 ) , ( 7.2 , 9.6 ) (2.4,7.2),(3,7.2),(7.2,9.6) }, with { ( 2.4 , 7.2 ) , ( 3 , 7.2 ) } \{(2.4,7.2),(3,7.2)\} yielding similar performance and ( 7.2 , 9.6 ) (7.2,9.6) slightly underperforming. For our experiments in Section 4.3 , for both BERT and T5, we choose ( 3 , 7.2 ) (3,7.2) , as it maintains the same stacking step all training steps \frac{\text{stacking step}}{\text{all training steps}} ratio proposed by Gong et al. [30] .

### C.2 Layer dropping : How to Drop

In Figure 8 , we observe that different choices of γ f \gamma_{f} yield comparable performance for both the BERT and T5 models. However, the layer dropping training curves for the T5 model are notably spikier than those of the baseline, suggesting that the layer dropping method demonstrates less stability during training with this architecture. As a result, we also tested a smaller learning rate ( 1 ​ e − 2 ) (1e-2) , in contrast to the original training rate ( 2 ​ e − 2 ) (2e-2) , which ultimately yielded better results. The parameters selected for our experiments in Section 4.3 were γ f = 20 , l ​ r = 1 ​ e − 2 \gamma_{f}=20,lr=1e-2 for T5 and γ f = 100 \gamma_{f}=100 for BERT.

### C.3 Selective backprop : Selectivity Scale

We tune the selectivity scale β \beta , where β ∈ { 1 , 2 , 3 } \beta\in\{1,2,3\} . Jiang et al. [38] use 33% and 50% selectivity in their experiments, which approximately corresponds to β = { 1 , 2 } \beta=\{1,2\} , respectively. We find that the larger the β \beta value, the worse the pre-training performance. Note that the higher the β \beta value, the more forward passes selective backprop needs to perform in order to collect enough samples for a backward pass, which decreases the total number of parameter update steps within the RST budget. For the experiments in Section 5.3 , we chose β = 1 \beta=1 , as it consistently achieves the best performance.

### C.4 RHO loss : Mega-batch Size

RHO loss requires one additional hyper-parameter, the size of the mega-batch, from which the mini-batch then gets subsampled from. We tune this hyper-parameter in Figure 10(a) and similar to Section C.3 , we find that the larger this size gets, the worse the validation loss. We started tuning it based on BCWK, and given the clear hierarchy we observed; we decided not to tune it on other datasets and simply set it to 2x (3072).

Another implicit set of hyper-parameters is how to pre-train the proxy/irreducible loss models. Here, we follow suggestions by Mindermann et al. [65] and choose the same architecture as the target model. We split all datasets into 20% proxy model pre-training, ~ 1% proxy model validation, and ~ 79% target model pre-training set (later, during target model pre-training, we further split the remaining 79% into train and validation set). For BCWK and C4, we train for 12 hours; for MP, we train for 6 hours only since it is much smaller and we want to avoid over-fitting. Other than varying the dataset splits and training budgets, we use the same hyper-parameters from Section 3 .

Figure 10(b) shows the validation loss during proxy model pre-training; none of the models over-fit, and all of them reach reasonable losses. For reference, one may compare their values with the baseline in Figure 9 , which shows the validation losses using data from the same dataset sources.

### C.5 Lion : Learning Rate (LR) And Weight Decay (WD)

The authors of Lion provide the following guidelines [ 13 , page 14] :

We adopt these default β 1 , β 2 \beta_{1},\beta_{2} hyper-parameters. Next, we look at LR and WD (also from [ 13 ] ).

To recap (details in Section 3 ), for AdamW, we use base LRs of 1e-3 and 0.02 0.02 , respectively. For BERT, we use a WD of 0.01 0.01 , while we disable it for T5 respectively.

Hence, following the above guidelines, we define the grid search space as described in Table 5 .

For BERT, we determine a LR of 7e-4 and a weight decay of 0.1 0.1 to be best, as illustrated in Figure 11(a) . For T5, we do not use any weight decay (following Raffel et al. [76] , Shazeer [81] ) and find an LR of 7.5e-4 to yield the best performance, as shown in Figure 11(b) . For all optimizers, we follow Nawrot [68] to integrate RMS-LR scaling to facilitate convergence.

### C.6 Sophia : Learning Rate (LR), Weight Decay (WD) and ρ \rho

The official code repository 4 4 4 https://github.com/Liuhong99/Sophia/blob/main/README.md suggests the following:

First, we followed the above guidelines, replaced AdamW with Sophia , and simply tuned ρ ∈ { 0.01 , 0.03 , 0.05 , 0.1 } \rho\in\{0.01,0.03,0.05,0.1\} . All of these runs led to NaN losses, which we illustrate in Figure 12 .

Next, we lowered the learning rate to 1e-4, which mitigated the instabilities but resulted in much slower convergence, as shown in Figure 13 .

We then decided to switch from FP16 to BF16, which improved training stability. Further, we manually tuned the LR, WD, and ρ \rho values, as shown in Figure 14 . Since we noticed a strong negative correlation between ρ \rho and the training loss, we decided to stick to ρ = 0.01 \rho=0.01 . The best performance was achieved with an LR of 4e-4 and a WD of 0.015 0.015 . We follow Nawrot [68] to integrate RMS-LR scaling to facilitate convergence, as for all optimizers.

For T5, we vary the learning rate within the range of { 2 ​ e − 2 , 5 ​ e − 3 , 2.5 ​ e − 3 , 1 ​ e − 3 , 7.5 ​ e − 4 , 5 ​ e − 4 } \{2e-2,5e-3,2.5e-3,1e-3,7.5e-4,5e-4\} , and ρ \rho in { 5 ​ e − 2 , 1 ​ e − 2 } \{5e-2,1e-2\} . Figure 15 show the results; similar to BERT, we observe better performance with a smaller ρ \rho . The best performance comes with ρ = 1 ​ e − 2 \rho=1e-2 and LR of 1e-3.

We acknowledge that there are additional hyper-parameters like ϵ \epsilon and k k that we did not tune because we followed the author’s suggestions. Future work may investigate their effects on the training speed-ups too.

## Appendix D Additional Results

### D.1 Training BERT, Evaluating on GLUE

We notice that on specific GLUE datasets, the efficiency methods have little effect on performance (MNLI, SST-2, QNLI, QQP, MRPC). Across all datasets, CoLA appears to have the largest gains from efficient training. Averaged across all datasets, efficient training methods have little effect. We report the exact numbers in Table 1 (left) for layer stacking and layer dropping . In these experiments, layer dropping barely outperforms the baseline given a 6-hour RST budget. For a 12-hour budget, both stacking and layer dropping inch above the baseline, and stacking only produces more accurate results than the baseline and layer dropping for a 24-hour budget. In our study, we also compare the performance of the efficient methods on the T5-base model evaluated on the SNI benchmark and report the results in Table 2 . From our observations, layer stacking is particularly noteworthy, demonstrating superior performance in a 6-hour training period, significantly outperforming the other methods. However, as the training budget is increased to 12 hours, the gap between baseline and layer stacking starts to diminish. In a 24-hour training scenario, baseline exhibits a marginally better performance than layer stacking , highlighting the efficacy of the baseline method given sufficient training time. Notably, both baseline and layer stacking consistently outperform layer dropping across all the time budgets.

### D.2 Layer dropping on small datasets

The layer dropping paper [ 106 ] trains on the Bookcorpus+Wikipedia dataset for 186 186 epochs, while the original BERT paper trains for only 40 epochs [ 22 ] . They do not report results for the baseline trained with a schedule based on fewer epochs. Given the possibility of overfitting due to the high number of epochs layer dropping ’s similarity to dropout [ 88 ] , and dropout’s known efficacy for language model pre-training with repeated data [ 103 ] , we raise the question of whether layer dropping acts as a regularizer.

Note that given the abundance of pre-training data for language models, even the largest and most-expensively-trained models [ 34 , 14 , 91 ] are typically trained within a one-epoch regime, which has been shown to improve performance over training for multiple epochs on a smaller dataset [ 50 , 76 ] . Hence, we exhaust the compute budget before passing through all available training data more than once. In contrast, Zhang & He [106] use a smaller dataset and thus complete 186 186 epochs during training. Thus, their training setup likely corresponds to an overfitting regime.

To investigate this, we repeat the experiment with a truncated training dataset, resulting in the experiment performing roughly 180 180 epochs. The result is shown in Figure 17(a) . The plot confirms our suspicion: when the baseline training procedure overfits, layer dropping can help mitigate this, preventing the validation loss from increasing as training time increases.

## Instructions for reporting errors

We are continuing to improve HTML versions of papers, and your feedback helps enhance accessibility and mobile support. To report errors in the HTML that will help us improve conversion and rendering, choose any of the methods listed below:

Click the "Report Issue" ( ) button, located in the page header.

Tip: You can select the relevant text first, to include it in your report.

Our team has already identified the following issues . We appreciate your time reviewing and reporting rendering errors we may not have found yet. Your efforts will help us improve the HTML versions for all readers, because disability should not be a barrier to accessing research. Thank you for your continued support in championing open access for all.

Have a free development cycle? Help support accessibility at arXiv! Our collaborators at LaTeXML maintain a list of packages that need conversion , and welcome developer contributions .