Diformer: Directional Transformer for Neural Machine Translation
This addresses the trade-off between latency and accuracy in machine translation, offering a competitive solution for real-time applications, though it is incremental in improving existing combination methods.
The paper tackles the performance gap in combining autoregressive (AR) and non-autoregressive (NAR) models for neural machine translation by proposing Diformer, which unifies them under a directional framework, resulting in over 1.5 BLEU point gains on WMT benchmarks compared to prior unified models.
Autoregressive (AR) and Non-autoregressive (NAR) models have their own superiority on the performance and latency, combining them into one model may take advantage of both. Current combination frameworks focus more on the integration of multiple decoding paradigms with a unified generative model, e.g. Masked Language Model. However, the generalization can be harmful to the performance due to the gap between training objective and inference. In this paper, we aim to close the gap by preserving the original objective of AR and NAR under a unified framework. Specifically, we propose the Directional Transformer (Diformer) by jointly modelling AR and NAR into three generation directions (left-to-right, right-to-left and straight) with a newly introduced direction variable, which works by controlling the prediction of each token to have specific dependencies under that direction. The unification achieved by direction successfully preserves the original dependency assumption used in AR and NAR, retaining both generalization and performance. Experiments on 4 WMT benchmarks demonstrate that Diformer outperforms current united-modelling works with more than 1.5 BLEU points for both AR and NAR decoding, and is also competitive to the state-of-the-art independent AR and NAR models.