Small Batch Sizes Improve Training of Low-Resource Neural MT
This work addresses hyperparameter optimization for low-resource neural machine translation, offering a practical improvement for researchers and practitioners in this domain.
The paper tackles the problem of training Transformers for neural machine translation in low-resource settings by challenging the common practice of using large batch sizes, showing that smaller batch sizes lead to higher scores and shorter training times.
We study the role of an essential hyper-parameter that governs the training of Transformers for neural machine translation in a low-resource setting: the batch size. Using theoretical insights and experimental evidence, we argue against the widespread belief that batch size should be set as large as allowed by the memory of the GPUs. We show that in a low-resource setting, a smaller batch size leads to higher scores in a shorter training time, and argue that this is due to better regularization of the gradients during training.