DeltaScore: Fine-Grained Story Evaluation with Perturbations
This addresses the need for better story evaluation metrics in natural language generation, though it is incremental as it builds on existing perturbation and language model techniques.
The paper tackled the problem of evaluating nuanced aspects of stories like fluency and interestingness by introducing DELTASCORE, a method using perturbations and pre-trained language models, which demonstrated remarkable performance on storytelling datasets across five fine-grained aspects.
Numerous evaluation metrics have been developed for natural language generation tasks, but their effectiveness in evaluating stories is limited as they are not specifically tailored to assess intricate aspects of storytelling, such as fluency and interestingness. In this paper, we introduce DELTASCORE, a novel methodology that employs perturbation techniques for the evaluation of nuanced story aspects. Our central proposition posits that the extent to which a story excels in a specific aspect (e.g., fluency) correlates with the magnitude of its susceptibility to particular perturbations (e.g., the introduction of typos). Given this, we measure the quality of an aspect by calculating the likelihood difference between pre- and post-perturbation states using pre-trained language models. We compare DELTASCORE with existing metrics on storytelling datasets from two domains in five fine-grained story aspects: fluency, coherence, relatedness, logicality, and interestingness. DELTASCORE demonstrates remarkable performance, revealing a surprising finding that a specific perturbation proves highly effective in capturing multiple aspects.