DePA: Improving Non-autoregressive Machine Translation with Dependency-Aware Decoder
This addresses the quality gap for users of NAT models in machine translation, offering an incremental improvement over existing methods.
The paper tackles the lower translation quality of non-autoregressive machine translation (NAT) models by proposing DePA, a dependency-aware decoder that improves target dependency modeling, resulting in up to 1.88 BLEU gain on benchmarks while maintaining inference latency.
Non-autoregressive machine translation (NAT) models have lower translation quality than autoregressive translation (AT) models because NAT decoders do not depend on previous target tokens in the decoder input. We propose a novel and general Dependency-Aware Decoder (DePA) to enhance target dependency modeling in the decoder of fully NAT models from two perspectives: decoder self-attention and decoder input. First, we propose an autoregressive forward-backward pre-training phase before NAT training, which enables the NAT decoder to gradually learn bidirectional target dependencies for the final NAT training. Second, we transform the decoder input from the source language representation space to the target language representation space through a novel attentive transformation process, which enables the decoder to better capture target dependencies. DePA can be applied to any fully NAT models. Extensive experiments show that DePA consistently improves highly competitive and state-of-the-art fully NAT models on widely used WMT and IWSLT benchmarks by up to 1.88 BLEU gain, while maintaining the inference latency comparable to other fully NAT models.