Sequence Generation: From Both Sides to the Middle
This addresses efficiency and quality issues in sequence generation for applications like machine translation and text summarization, representing a novel method rather than an incremental improvement.
The paper tackles the slow decoding and lack of future context in autoregressive sequence generation by proposing a synchronous bidirectional model that generates from both sides to the middle, which significantly speeds up decoding and improves quality on translation and summarization tasks.
The encoder-decoder framework has achieved promising process for many sequence generation tasks, such as neural machine translation and text summarization. Such a framework usually generates a sequence token by token from left to right, hence (1) this autoregressive decoding procedure is time-consuming when the output sentence becomes longer, and (2) it lacks the guidance of future context which is crucial to avoid under translation. To alleviate these issues, we propose a synchronous bidirectional sequence generation (SBSG) model which predicts its outputs from both sides to the middle simultaneously. In the SBSG model, we enable the left-to-right (L2R) and right-to-left (R2L) generation to help and interact with each other by leveraging interactive bidirectional attention network. Experiments on neural machine translation (En-De, Ch-En, and En-Ro) and text summarization tasks show that the proposed model significantly speeds up decoding while improving the generation quality compared to the autoregressive Transformer.