Understanding Factuality in Abstractive Summarization with FRANK: A Benchmark for Factuality Metrics
This work addresses the lack of common benchmarks for comparing factuality metrics in summarization, providing deeper insights into error types, but it is incremental as it builds on existing datasets and models.
The paper tackles the problem of factuality in abstractive summarization by creating FRANK, a benchmark that categorizes factual errors and uses human annotations to evaluate summarization models and factuality metrics on CNN/DM and XSum datasets, showing correlations with human judgment and identifying error proportions.
Modern summarization models generate highly fluent but often factually unreliable outputs. This motivated a surge of metrics attempting to measure the factuality of automatically generated summaries. Due to the lack of common benchmarks, these metrics cannot be compared. Moreover, all these methods treat factuality as a binary concept and fail to provide deeper insights into the kinds of inconsistencies made by different systems. To address these limitations, we devise a typology of factual errors and use it to collect human annotations of generated summaries from state-of-the-art summarization systems for the CNN/DM and XSum datasets. Through these annotations, we identify the proportion of different categories of factual errors in various summarization models and benchmark factuality metrics, showing their correlation with human judgment as well as their specific strengths and weaknesses.