What Makes a Good Story and How Can We Measure It? A Comprehensive Survey of Story Evaluation
This survey addresses the problem of story evaluation for researchers and practitioners in AI and NLP, but it is incremental as it organizes existing knowledge rather than introducing new methods.
The paper tackles the challenge of automatically evaluating story quality, which is more complex than other generation tasks due to the need for measures like coherence and character development, by providing a comprehensive survey that summarizes storytelling tasks, proposes a taxonomy for evaluation metrics, and discusses human-AI collaboration.
With the development of artificial intelligence, particularly the success of Large Language Models (LLMs), the quantity and quality of automatically generated stories have significantly increased. This has led to the need for automatic story evaluation to assess the generative capabilities of computing systems and analyze the quality of both automatic-generated and human-written stories. Evaluating a story can be more challenging than other generation evaluation tasks. While tasks like machine translation primarily focus on assessing the aspects of fluency and accuracy, story evaluation demands complex additional measures such as overall coherence, character development, interestingness, etc. This requires a thorough review of relevant research. In this survey, we first summarize existing storytelling tasks, including text-to-text, visual-to-text, and text-to-visual. We highlight their evaluation challenges, identify various human criteria to measure stories, and present existing benchmark datasets. Then, we propose a taxonomy to organize evaluation metrics that have been developed or can be adopted for story evaluation. We also provide descriptions of these metrics, along with the discussion of their merits and limitations. Later, we discuss the human-AI collaboration for story evaluation and generation. Finally, we suggest potential future research directions, extending from story evaluation to general evaluations.