Improving Non-autoregressive Generation with Mixup Training
This addresses the problem of slow inference in non-autoregressive generation for NLP applications, offering a method that maintains speed while improving performance, though it is incremental as it builds on existing pre-trained models.
The paper tackles the challenge of leveraging pre-trained language models for non-autoregressive generation by proposing MIST, a training method that bridges the gap with autoregressive models without affecting inference speed, achieving state-of-the-art results on benchmarks like question generation, summarization, and paraphrase generation.
While pre-trained language models have achieved great success on various natural language understanding tasks, how to effectively leverage them into non-autoregressive generation tasks remains a challenge. To solve this problem, we present a non-autoregressive generation model based on pre-trained transformer models. To bridge the gap between autoregressive and non-autoregressive models, we propose a simple and effective iterative training method called MIx Source and pseudo Target (MIST). Unlike other iterative decoding methods, which sacrifice the inference speed to achieve better performance based on multiple decoding iterations, MIST works in the training stage and has no effect on inference time. Our experiments on three generation benchmarks including question generation, summarization and paraphrase generation, show that the proposed framework achieves the new state-of-the-art results for fully non-autoregressive models. We also demonstrate that our method can be used to a variety of pre-trained models. For instance, MIST based on the small pre-trained model also obtains comparable performance with seq2seq models.