Learning Non-Autoregressive Models from Search for Unsupervised Sentence Summarization
This addresses the problem of generating summaries without parallel data for researchers and practitioners in NLP, though it is incremental as it builds on existing non-autoregressive and unsupervised methods.
The paper tackles unsupervised sentence summarization by proposing a non-autoregressive model trained from edit-based search results, achieving state-of-the-art performance on two datasets while significantly improving inference efficiency.
Text summarization aims to generate a short summary for an input text. In this work, we propose a Non-Autoregressive Unsupervised Summarization (NAUS) approach, which does not require parallel data for training. Our NAUS first performs edit-based search towards a heuristically defined score, and generates a summary as pseudo-groundtruth. Then, we train an encoder-only non-autoregressive Transformer based on the search result. We also propose a dynamic programming approach for length-control decoding, which is important for the summarization task. Experiments on two datasets show that NAUS achieves state-of-the-art performance for unsupervised summarization, yet largely improving inference efficiency. Further, our algorithm is able to perform explicit length-transfer summary generation.