How to Evaluate Coreference in Literary Texts?
This addresses the challenge of evaluating coreference resolution specifically for literary analysis, particularly in novels, though it appears incremental as it builds on existing metrics with a domain-specific refinement.
The paper tackles the problem of evaluating coreference in literary texts by showing that existing single-score metrics are uninformative or misleading, and proposes a new evaluation approach that distinguishes between long coreference chains for main characters, short chains for secondary characters, and singletons for isolated elements to yield more interpretable results.
In this short paper, we examine the main metrics used to evaluate textual coreference and we detail some of their limitations. We show that a unique score cannot represent the full complexity of the problem at stake, and is thus uninformative, or even misleading. We propose a new way of evaluating coreference, taking into account the context (in our case, the analysis of fictions, esp. novels). More specifically, we propose to distinguish long coreference chains (corresponding to main characters), from short ones (corresponding to secondary characters), and singletons (isolated elements). This way, we hope to get more interpretable and thus more informative results through evaluation.