FEQA: A Question Answering Evaluation Framework for Faithfulness Assessment in Abstractive Summarization
This addresses the issue of unfaithful summaries for users relying on automated summarization tools, though it is incremental as it builds on existing QA methods for evaluation.
The paper tackles the problem of evaluating faithfulness in abstractive summarization, where models often generate content inconsistent with the source document, and proposes FEQA, a question answering-based metric that shows significantly higher correlation with human scores, especially for highly abstractive summaries.
Neural abstractive summarization models are prone to generate content inconsistent with the source document, i.e. unfaithful. Existing automatic metrics do not capture such mistakes effectively. We tackle the problem of evaluating faithfulness of a generated summary given its source document. We first collected human annotations of faithfulness for outputs from numerous models on two datasets. We find that current models exhibit a trade-off between abstractiveness and faithfulness: outputs with less word overlap with the source document are more likely to be unfaithful. Next, we propose an automatic question answering (QA) based metric for faithfulness, FEQA, which leverages recent advances in reading comprehension. Given question-answer pairs generated from the summary, a QA model extracts answers from the document; non-matched answers indicate unfaithful information in the summary. Among metrics based on word overlap, embedding similarity, and learned language understanding models, our QA-based metric has significantly higher correlation with human faithfulness scores, especially on highly abstractive summaries.