No Train No Gain: Revisiting Efficient Training Algorithms For Transformer-based Language Models
This work addresses the problem of verifying the effectiveness of efficient training methods for large language models, revealing that many claimed improvements may be incremental or not robust under standardized evaluation.
The study re-evaluated efficient training algorithms for Transformer-based language models, such as dynamic architectures, batch selection, and efficient optimizers, and found that under a fixed computation budget, their gains in training, validation, and downstream performance disappeared compared to a baseline with a fully-decayed learning rate.
The computation necessary for training Transformer-based language models has skyrocketed in recent years. This trend has motivated research on efficient training algorithms designed to improve training, validation, and downstream performance faster than standard training. In this work, we revisit three categories of such algorithms: dynamic architectures (layer stacking, layer dropping), batch selection (selective backprop, RHO loss), and efficient optimizers (Lion, Sophia). When pre-training BERT and T5 with a fixed computation budget using such methods, we find that their training, validation, and downstream gains vanish compared to a baseline with a fully-decayed learning rate. We define an evaluation protocol that enables computation to be done on arbitrary machines by mapping all computation time to a reference machine which we call reference system time. We discuss the limitations of our proposed protocol and release our code to encourage rigorous research in efficient training procedures: https://github.com/JeanKaddour/NoTrainNoGain.