Towards Coherent and Consistent Use of Entities in Narrative Generation
This addresses a specific issue in narrative generation for improving story quality, but it is incremental as it builds on existing language models with a novel augmentation.
The paper tackles the problem of maintaining entity coherence and consistency in narrative generation by large language models, proposing a dynamic entity memory that improves entity coherence according to automatic metrics and human judgment, with validation showing the metrics correlate with human ratings.
Large pre-trained language models (LMs) have demonstrated impressive capabilities in generating long, fluent text; however, there is little to no analysis on their ability to maintain entity coherence and consistency. In this work, we focus on the end task of narrative generation and systematically analyse the long-range entity coherence and consistency in generated stories. First, we propose a set of automatic metrics for measuring model performance in terms of entity usage. Given these metrics, we quantify the limitations of current LMs. Next, we propose augmenting a pre-trained LM with a dynamic entity memory in an end-to-end manner by using an auxiliary entity-related loss for guiding the reads and writes to the memory. We demonstrate that the dynamic entity memory increases entity coherence according to both automatic and human judgment and helps preserving entity-related information especially in settings with a limited context window. Finally, we also validate that our automatic metrics are correlated with human ratings and serve as a good indicator of the quality of generated stories.