Metrics also Disagree in the Low Scoring Range: Revisiting Summarization Evaluation Metrics
This addresses the problem of unreliable automatic evaluation in text summarization for researchers, though it is incremental as it builds on existing findings.
The paper revisits prior work on summarization evaluation metrics, finding that metrics disagree across all scoring ranges, not just high-scoring ones, and analyzes properties like Ease of Summarization, Abstractiveness, and Coverage that affect this agreement.
In text summarization, evaluating the efficacy of automatic metrics without human judgments has become recently popular. One exemplar work concludes that automatic metrics strongly disagree when ranking high-scoring summaries. In this paper, we revisit their experiments and find that their observations stem from the fact that metrics disagree in ranking summaries from any narrow scoring range. We hypothesize that this may be because summaries are similar to each other in a narrow scoring range and are thus, difficult to rank. Apart from the width of the scoring range of summaries, we analyze three other properties that impact inter-metric agreement - Ease of Summarization, Abstractiveness, and Coverage. To encourage reproducible research, we make all our analysis code and data publicly available.