Future-Guided Incremental Transformer for Simultaneous Translation
This work provides an incremental improvement to the efficiency and quality of simultaneous translation for users in online scenarios.
This paper addresses the challenges of low training speed and lack of future source information in the wait-k policy for simultaneous translation. They propose an incremental Transformer with an average embedding layer (AEL) to accelerate training by approximately 28 times and use knowledge distillation from a conventional Transformer to embed future information, resulting in improved translation quality over the wait-k baseline.
Simultaneous translation (ST) starts translations synchronously while reading source sentences, and is used in many online scenarios. The previous wait-k policy is concise and achieved good results in ST. However, wait-k policy faces two weaknesses: low training speed caused by the recalculation of hidden states and lack of future source information to guide training. For the low training speed, we propose an incremental Transformer with an average embedding layer (AEL) to accelerate the speed of calculation of the hidden states during training. For future-guided training, we propose a conventional Transformer as the teacher of the incremental Transformer, and try to invisibly embed some future information in the model through knowledge distillation. We conducted experiments on Chinese-English and German-English simultaneous translation tasks and compared with the wait-k policy to evaluate the proposed method. Our method can effectively increase the training speed by about 28 times on average at different k and implicitly embed some predictive abilities in the model, achieving better translation quality than wait-k baseline.