Optimizing Non-Autoregressive Transformers with Contrastive Learning
This work addresses a key bottleneck in NATs for applications like machine translation, offering significant performance improvements, though it is incremental as it builds on existing NAT architectures.
The paper tackles the challenge of learning multi-modality data distributions in Non-Autoregressive Transformers (NATs), which causes a performance gap with Autoregressive Transformers, by proposing a method that samples from the model distribution and uses contrastive constraints to stabilize training, resulting in new state-of-the-art results on benchmarks for machine translation, text summarization, and paraphrasing.
Non-autoregressive Transformers (NATs) reduce the inference latency of Autoregressive Transformers (ATs) by predicting words all at once rather than in sequential order. They have achieved remarkable progress in machine translation as well as many other applications. However, a long-standing challenge for NATs is the learning of multi-modality data distribution, which is the main cause of the performance gap between NATs and ATs. In this paper, we propose to ease the difficulty of modality learning via sampling from the model distribution instead of the data distribution. We derive contrastive constraints to stabilize the training process and integrate this resulting objective with the state-of-the-art NAT architecture DA-Transformer. Our model \method is examined on 3 different tasks, including machine translation, text summarization, and paraphrasing with 5 benchmarks. Results show that our approach outperforms previous non-autoregressive baselines by a significant margin and establishes new state-of-the-art results for non-autoregressive transformers on all the benchmarks.