Towards Coherent and Cohesive Long-form Text Generation
This work addresses the challenge of long-form text generation for natural language processing applications, but it is incremental as it builds on existing neural language models with specific enhancements.
The paper tackled the problem of generating coherent and cohesive long-form text by proposing a neural language model with discriminators for sentence and paragraph feedback, trained using negative-critical sequence training, resulting in improvements over a strong baseline model.
Generating coherent and cohesive long-form texts is a challenging task. Previous works relied on large amounts of human-generated texts to train neural language models. However, few attempted to explicitly improve neural language models from the perspectives of coherence and cohesion. In this work, we propose a new neural language model that is equipped with two neural discriminators which provide feedback signals at the levels of sentence (cohesion) and paragraph (coherence). Our model is trained using a simple yet efficient variant of policy gradient, called negative-critical sequence training, which is proposed to eliminate the need of training a separate critic for estimating baseline. Results demonstrate the effectiveness of our approach, showing improvements over the strong baseline -- recurrent attention-based bidirectional MLE-trained neural language model.