A Survey on Efficient Training of Transformers
It addresses the high resource demands of Transformer training, which is a bottleneck for researchers and practitioners, but is incremental as it compiles existing techniques.
This survey systematically reviews methods to reduce the computational and memory costs of training Transformers, aiming to make training faster, cheaper, and more accurate.
Recent advances in Transformers have come with a huge requirement on computing resources, highlighting the importance of developing efficient training techniques to make Transformer training faster, at lower cost, and to higher accuracy by the efficient use of computation and memory resources. This survey provides the first systematic overview of the efficient training of Transformers, covering the recent progress in acceleration arithmetic and hardware, with a focus on the former. We analyze and compare methods that save computation and memory costs for intermediate tensors during training, together with techniques on hardware/algorithm co-design. We finally discuss challenges and promising areas for future research.