Is human scoring the best criteria for summary evaluation?
This work addresses a foundational problem in natural language processing for researchers developing and evaluating summary quality metrics, suggesting a potentially more robust evaluation method.
This paper questions the common practice of using human scores as the sole criterion for evaluating summary quality measures. It proposes an alternative, human-score-independent criterion for selecting the best measure within a family of measures, demonstrating its universality across different summary styles for the BLANC family.
Normally, summary quality measures are compared with quality scores produced by human annotators. A higher correlation with human scores is considered to be a fair indicator of a better measure. We discuss observations that cast doubt on this view. We attempt to show a possibility of an alternative indicator. Given a family of measures, we explore a criterion of selecting the best measure not relying on correlations with human scores. Our observations for the BLANC family of measures suggest that the criterion is universal across very different styles of summaries.