Training Tips for the Transformer Model
This work offers incremental improvements for researchers training Transformer models by addressing practical bottlenecks like memory usage and training stability.
The authors investigated how key parameters affect the performance and efficiency of the Transformer model in neural machine translation, providing practical recommendations for optimizing training based on experiments with data, model size, and hardware constraints.
This article describes our experiments in neural machine translation using the recent Tensor2Tensor framework and the Transformer sequence-to-sequence model (Vaswani et al., 2017). We examine some of the critical parameters that affect the final translation quality, memory usage, training stability and training time, concluding each experiment with a set of recommendations for fellow researchers. In addition to confirming the general mantra "more data and larger models", we address scaling to multiple GPUs and provide practical tips for improved training regarding batch size, learning rate, warmup steps, maximum sentence length and checkpoint averaging. We hope that our observations will allow others to get better results given their particular hardware and data constraints.