Optimizing transformer-based machine translation model for single GPU training: a hyperparameter ablation study
This work addresses the problem of high computational costs for machine translation researchers and practitioners, offering incremental insights for more accessible and cost-effective training.
This study tackled the assumption that larger models always perform better in machine translation by conducting a hyperparameter ablation on a single GPU, finding that smaller parameter combinations could achieve similar translation quality without requiring multiple GPUs.
In machine translation tasks, the relationship between model complexity and performance is often presumed to be linear, driving an increase in the number of parameters and consequent demands for computational resources like multiple GPUs. To explore this assumption, this study systematically investigates the effects of hyperparameters through ablation on a sequence-to-sequence machine translation pipeline, utilizing a single NVIDIA A100 GPU. Contrary to expectations, our experiments reveal that combinations with the most parameters were not necessarily the most effective. This unexpected insight prompted a careful reduction in parameter sizes, uncovering "sweet spots" that enable training sophisticated models on a single GPU without compromising translation quality. The findings demonstrate an intricate relationship between hyperparameter selection, model size, and computational resource needs. The insights from this study contribute to the ongoing efforts to make machine translation more accessible and cost-effective, emphasizing the importance of precise hyperparameter tuning over mere scaling.