Decoding Methods for Neural Narrative Generation
This work addresses narrative generation for NLP applications, but it is incremental as it adapts existing methods to a similar task.
The study tackled the problem of applying decoding methods from neural response generation to narrative generation, finding that nucleus sampling with thresholds of 0.7-0.9 works best, maximum mutual information improves story quality, and automatic metrics poorly correlate with human judgments.
Narrative generation is an open-ended NLP task in which a model generates a story given a prompt. The task is similar to neural response generation for chatbots; however, innovations in response generation are often not applied to narrative generation, despite the similarity between these tasks. We aim to bridge this gap by applying and evaluating advances in decoding methods for neural response generation to neural narrative generation. In particular, we employ GPT-2 and perform ablations across nucleus sampling thresholds and diverse decoding hyperparameters -- specifically, maximum mutual information -- analyzing results over multiple criteria with automatic and human evaluation. We find that (1) nucleus sampling is generally best with thresholds between 0.7 and 0.9; (2) a maximum mutual information objective can improve the quality of generated stories; and (3) established automatic metrics do not correlate well with human judgments of narrative quality on any qualitative metric.