PAIR: Planning and Iterative Refinement in Pre-trained Transformers for Long Text Generation
This addresses the issue of rambling text generation for users of large language models, though it is incremental as it builds on existing models like BART.
The authors tackled the problem of generating coherent long text by introducing PAIR, a content-controlled framework that uses planning and iterative refinement with pre-trained Transformers, resulting in average improvements of 20 BLEU and 12 METEOR points across three domains.
Pre-trained Transformers have enabled impressive breakthroughs in generating long and fluent text, yet their outputs are often "rambling" without coherently arranged content. In this work, we present a novel content-controlled text generation framework, PAIR, with planning and iterative refinement, which is built upon a large model, BART. We first adapt the BERT model to automatically construct the content plans, consisting of keyphrase assignments and their corresponding sentence-level positions. The BART model is employed for generation without modifying its structure. We then propose a refinement algorithm to gradually enhance the generation quality within the sequence-to-sequence framework. Evaluation with automatic metrics shows that adding planning consistently improves the generation quality on three distinct domains, with an average of 20 BLEU points and 12 METEOR points improvements. In addition, human judges rate our system outputs to be more relevant and coherent than comparisons without planning.