Evaluation of Automatic Text Summarization using Synthetic Facts
This addresses evaluation and factual consistency issues in text summarization for applications, but it appears incremental as it builds on known problems without claiming broad SOTA impact.
The authors tackled the problem of unreliable automatic text summarization by proposing a new reference-less evaluation system that measures quality based on factual consistency, comprehensiveness, and compression rate, but no concrete numbers are provided in the abstract.
Despite some recent advances, automatic text summarization remains unreliable, elusive, and of limited practical use in applications. Two main problems with current summarization methods are well known: evaluation and factual consistency. To address these issues, we propose a new automatic reference-less text summarization evaluation system that can measure the quality of any text summarization model with a set of generated facts based on factual consistency, comprehensiveness, and compression rate. As far as we know, our evaluation system is the first system that measures the overarching quality of the text summarization models based on factuality, information coverage, and compression rate.