Evaluating Computational Representations of Character: An Austen Character Similarity Benchmark
This work addresses the need for better evaluation benchmarks in computational literary analysis, though it is incremental as it builds on existing pipelines and focuses on a specific domain (Jane Austen's novels).
The authors tackled the problem of evaluating computational representations of literary characters by proposing character similarity grouping as a holistic task, and they found that while systems like BookNLP and FanfictionNLP capture broad similarities, expert-defined pairings remain challenging, highlighting gaps in capturing subtle human-noted aspects.
Several systems have been developed to extract information about characters to aid computational analysis of English literature. We propose character similarity grouping as a holistic evaluation task for these pipelines. We present AustenAlike, a benchmark suite of character similarities in Jane Austen's novels. Our benchmark draws on three notions of character similarity: a structurally defined notion of similarity; a socially defined notion of similarity; and an expert defined set extracted from literary criticism. We use AustenAlike to evaluate character features extracted using two pipelines, BookNLP and FanfictionNLP. We build character representations from four kinds of features and compare them to the three AustenAlike benchmarks and to GPT-4 similarity rankings. We find that though computational representations capture some broad similarities based on shared social and narrative roles, the expert pairings in our third benchmark are challenging for all systems, highlighting the subtler aspects of similarity noted by human readers.