Questioning the Validity of Summarization Datasets and Improving Their Factual Consistency
This addresses dataset validity issues for researchers and developers in NLP, though it is incremental as it builds on existing consistency models.
The authors identified that popular abstractive summarization datasets lack factual consistency and used state-of-the-art models to filter them, releasing SummFC, which improved model performance across nearly all quality aspects.
The topic of summarization evaluation has recently attracted a surge of attention due to the rapid development of abstractive summarization systems. However, the formulation of the task is rather ambiguous, neither the linguistic nor the natural language processing community has succeeded in giving a mutually agreed-upon definition. Due to this lack of well-defined formulation, a large number of popular abstractive summarization datasets are constructed in a manner that neither guarantees validity nor meets one of the most essential criteria of summarization: factual consistency. In this paper, we address this issue by combining state-of-the-art factual consistency models to identify the problematic instances present in popular summarization datasets. We release SummFC, a filtered summarization dataset with improved factual consistency, and demonstrate that models trained on this dataset achieve improved performance in nearly all quality aspects. We argue that our dataset should become a valid benchmark for developing and evaluating summarization systems.