RoViST:Learning Robust Metrics for Visual Storytelling
This addresses the need for better evaluation in visual storytelling, providing more reliable metrics for researchers and developers, though it is incremental as it builds on existing evaluation frameworks.
The paper tackled the problem of evaluating visual storytelling models by proposing three new metric sets for visual grounding, coherence, and non-redundancy, which outperformed traditional metrics like BLEU in correlation with human judgment scores on stories from four state-of-the-art models.
Visual storytelling (VST) is the task of generating a story paragraph that describes a given image sequence. Most existing storytelling approaches have evaluated their models using traditional natural language generation metrics like BLEU or CIDEr. However, such metrics based on n-gram matching tend to have poor correlation with human evaluation scores and do not explicitly consider other criteria necessary for storytelling such as sentence structure or topic coherence. Moreover, a single score is not enough to assess a story as it does not inform us about what specific errors were made by the model. In this paper, we propose 3 evaluation metrics sets that analyses which aspects we would look for in a good story: 1) visual grounding, 2) coherence, and 3) non-redundancy. We measure the reliability of our metric sets by analysing its correlation with human judgement scores on a sample of machine stories obtained from 4 state-of-the-arts models trained on the Visual Storytelling Dataset (VIST). Our metric sets outperforms other metrics on human correlation, and could be served as a learning based evaluation metric set that is complementary to existing rule-based metrics.