Dynamic Multi-Branch Layers for On-Device Neural Machine Translation
This addresses the challenge of deploying efficient and accurate machine translation on mobile devices, representing an incremental improvement in model efficiency.
The paper tackles the problem of improving on-device neural machine translation performance under hardware constraints by proposing dynamic multi-branch layers, achieving up to 1.7 BLEU points improvement on English-German and 1.8 BLEU points on Chinese-English tasks over the Transformer model.
With the rapid development of artificial intelligence (AI), there is a trend in moving AI applications, such as neural machine translation (NMT), from cloud to mobile devices. Constrained by limited hardware resources and battery, the performance of on-device NMT systems is far from satisfactory. Inspired by conditional computation, we propose to improve the performance of on-device NMT systems with dynamic multi-branch layers. Specifically, we design a layer-wise dynamic multi-branch network with only one branch activated during training and inference. As not all branches are activated during training, we propose shared-private reparameterization to ensure sufficient training for each branch. At almost the same computational cost, our method achieves improvements of up to 1.7 BLEU points on the WMT14 English-German translation task and 1.8 BLEU points on the WMT20 Chinese-English translation task over the Transformer model, respectively. Compared with a strong baseline that also uses multiple branches, the proposed method is up to 1.5 times faster with the same number of parameters.