Efficiency through Auto-Sizing: Notre Dame NLP's Submission to the WNGT 2019 Efficiency Task
This work addresses efficiency in neural machine translation for researchers and practitioners, but it is incremental as it applies an existing auto-sizing method to a specific model.
The paper tackled the problem of reducing parameters in Transformer networks for machine translation, achieving a reduction of over 25% in parameters with only a 1.1 BLEU score decrease.
This paper describes the Notre Dame Natural Language Processing Group's (NDNLP) submission to the WNGT 2019 shared task (Hayashi et al., 2019). We investigated the impact of auto-sizing (Murray and Chiang, 2015; Murray et al., 2019) to the Transformer network (Vaswani et al., 2017) with the goal of substantially reducing the number of parameters in the model. Our method was able to eliminate more than 25% of the model's parameters while suffering a decrease of only 1.1 BLEU.