QURIOUS: Question Generation Pretraining for Text Generation
This work addresses the challenge of enhancing text generation quality for NLP applications, offering a novel pretraining approach that yields measurable improvements in specific tasks.
The paper tackles the problem of improving text generation by proposing question generation as a pretraining method, which better aligns with text generation objectives than task-agnostic approaches. The result is state-of-the-art performance on abstractive summarization and answer-focused question generation tasks, with human evaluators rating the outputs as more natural, concise, and informative.
Recent trends in natural language processing using pretraining have shifted focus towards pretraining and fine-tuning approaches for text generation. Often the focus has been on task-agnostic approaches that generalize the language modeling objective. We propose question generation as a pretraining method, which better aligns with the text generation objectives. Our text generation models pretrained with this method are better at understanding the essence of the input and are better language models for the target task. When evaluated on two text generation tasks, abstractive summarization and answer-focused question generation, our models result in state-of-the-art performances in terms of automatic metrics. Human evaluators also found our summaries and generated questions to be more natural, concise and informative.