Dynamically Adjusting Transformer Batch Size by Monitoring Gradient Direction Change
This work addresses the challenge of hyperparameter tuning for batch size in training large models like Transformers, offering an incremental improvement for machine translation efficiency.
The paper tackles the problem of optimizing batch size in neural models by analyzing gradient direction changes, proposing a method to dynamically adjust batch sizes during training, resulting in BLEU score improvements of +0.73 and +0.82 on WMT 14 translation tasks compared to a fixed batch size.
The choice of hyper-parameters affects the performance of neural models. While much previous research (Sutskever et al., 2013; Duchi et al., 2011; Kingma and Ba, 2015) focuses on accelerating convergence and reducing the effects of the learning rate, comparatively few papers concentrate on the effect of batch size. In this paper, we analyze how increasing batch size affects gradient direction, and propose to evaluate the stability of gradients with their angle change. Based on our observations, the angle change of gradient direction first tends to stabilize (i.e. gradually decrease) while accumulating mini-batches, and then starts to fluctuate. We propose to automatically and dynamically determine batch sizes by accumulating gradients of mini-batches and performing an optimization step at just the time when the direction of gradients starts to fluctuate. To improve the efficiency of our approach for large models, we propose a sampling approach to select gradients of parameters sensitive to the batch size. Our approach dynamically determines proper and efficient batch sizes during training. In our experiments on the WMT 14 English to German and English to French tasks, our approach improves the Transformer with a fixed 25k batch size by +0.73 and +0.82 BLEU respectively.