Unsupervised Text Generation by Learning from Search
This addresses the problem of generating high-quality text without labeled data for tasks like paraphrase generation and text formalization, representing an incremental advancement in unsupervised methods.
The paper tackles unsupervised text generation by alternating between search algorithms and generative models to improve sentence quality, achieving comparable performance to state-of-the-art supervised methods in paraphrase generation.
In this work, we present TGLS, a novel framework to unsupervised Text Generation by Learning from Search. We start by applying a strong search algorithm (in particular, simulated annealing) towards a heuristically defined objective that (roughly) estimates the quality of sentences. Then, a conditional generative model learns from the search results, and meanwhile smooth out the noise of search. The alternation between search and learning can be repeated for performance bootstrapping. We demonstrate the effectiveness of TGLS on two real-world natural language generation tasks, paraphrase generation and text formalization. Our model significantly outperforms unsupervised baseline methods in both tasks. Especially, it achieves comparable performance with the state-of-the-art supervised methods in paraphrase generation.