Predicting the Quality of Short Narratives from Social Media
This addresses the challenge of narrative quality evaluation for computational models, though it is incremental as it adapts existing methods to a new domain.
The authors tackled the problem of automatically evaluating narrative quality by using upvotes from social media as a proxy, collecting 54,484 answers from Quora and labeling 28,320 as stories to predict upvotes with neural networks that model textual regions and their interdependence.
An important and difficult challenge in building computational models for narratives is the automatic evaluation of narrative quality. Quality evaluation connects narrative understanding and generation as generation systems need to evaluate their own products. To circumvent difficulties in acquiring annotations, we employ upvotes in social media as an approximate measure for story quality. We collected 54,484 answers from a crowd-powered question-and-answer website, Quora, and then used active learning to build a classifier that labeled 28,320 answers as stories. To predict the number of upvotes without the use of social network features, we create neural networks that model textual regions and the interdependence among regions, which serve as strong benchmarks for future research. To our best knowledge, this is the first large-scale study for automatic evaluation of narrative quality.