FRSUM: Towards Faithful Abstractive Summarization via Enhancing Factual Robustness
This addresses the issue of generating inaccurate summaries in NLP, offering an incremental improvement for enhancing reliability in summarization systems.
The paper tackles the problem of unfaithful generation in abstractive summarization by introducing factual robustness as a measure and proposing FRSUM, a training strategy that improves faithfulness by defending against adversarial attacks, achieving consistent gains across models like T5 and BART.
Despite being able to generate fluent and grammatical text, current Seq2Seq summarization models still suffering from the unfaithful generation problem. In this paper, we study the faithfulness of existing systems from a new perspective of factual robustness which is the ability to correctly generate factual information over adversarial unfaithful information. We first measure a model's factual robustness by its success rate to defend against adversarial attacks when generating factual information. The factual robustness analysis on a wide range of current systems shows its good consistency with human judgments on faithfulness. Inspired by these findings, we propose to improve the faithfulness of a model by enhancing its factual robustness. Specifically, we propose a novel training strategy, namely FRSUM, which teaches the model to defend against both explicit adversarial samples and implicit factual adversarial perturbations. Extensive automatic and human evaluation results show that FRSUM consistently improves the faithfulness of various Seq2Seq models, such as T5, BART.