Auto-Sizing the Transformer Network: Improving Speed, Efficiency, and Performance for Low-Resource Machine Translation
This addresses the computational cost and performance issues for low-resource machine translation tasks, representing an incremental improvement over existing methods.
The paper tackled the problem of expensive architecture and hyperparameter optimization for Transformer models in low-resource machine translation by introducing auto-sizing, which removes neurons during training, resulting in up to a 3.9 BLEU score improvement and a one-third reduction in parameters.
Neural sequence-to-sequence models, particularly the Transformer, are the state of the art in machine translation. Yet these neural networks are very sensitive to architecture and hyperparameter settings. Optimizing these settings by grid or random search is computationally expensive because it requires many training runs. In this paper, we incorporate architecture search into a single training run through auto-sizing, which uses regularization to delete neurons in a network over the course of training. On very low-resource language pairs, we show that auto-sizing can improve BLEU scores by up to 3.9 points while removing one-third of the parameters from the model.