Learning to Recover from Multi-Modality Errors for Non-Autoregressive Neural Machine Translation
This addresses translation speed and accuracy issues for users of neural machine translation systems, representing an incremental improvement over existing non-autoregressive methods.
The paper tackled the multi-modality problem in non-autoregressive neural machine translation, which causes token repetitions or missing, by proposing RecoverSAT, a semi-autoregressive model that generates segments simultaneously while predicting tokens within each segment token-by-token, achieving over 4x speedup while maintaining comparable performance to autoregressive models on three benchmark datasets.
Non-autoregressive neural machine translation (NAT) predicts the entire target sequence simultaneously and significantly accelerates inference process. However, NAT discards the dependency information in a sentence, and thus inevitably suffers from the multi-modality problem: the target tokens may be provided by different possible translations, often causing token repetitions or missing. To alleviate this problem, we propose a novel semi-autoregressive model RecoverSAT in this work, which generates a translation as a sequence of segments. The segments are generated simultaneously while each segment is predicted token-by-token. By dynamically determining segment length and deleting repetitive segments, RecoverSAT is capable of recovering from repetitive and missing token errors. Experimental results on three widely-used benchmark datasets show that our proposed model achieves more than 4$\times$ speedup while maintaining comparable performance compared with the corresponding autoregressive model.