Entity-level Factual Adaptiveness of Fine-tuning based Abstractive Summarization Models
This work addresses factual inconsistency in summarization models, which is a critical issue for users relying on accurate summaries, but it is incremental as it builds on existing fine-tuning methods.
The paper tackled the problem of abstractive summarization models generating factually inconsistent content due to conflicts between model knowledge and input documents, and introduced a controllable counterfactual data augmentation method that improved factual adaptiveness while maintaining factual consistency on original datasets, achieving results comparable to a contrastive learning baseline.
Abstractive summarization models often generate factually inconsistent content particularly when the parametric knowledge of the model conflicts with the knowledge in the input document. In this paper, we analyze the robustness of fine-tuning based summarization models to the knowledge conflict, which we call factual adaptiveness. We utilize pre-trained language models to construct evaluation sets and find that factual adaptiveness is not strongly correlated with factual consistency on original datasets. Furthermore, we introduce a controllable counterfactual data augmentation method where the degree of knowledge conflict within the augmented data can be adjustable. Our experimental results on two pre-trained language models (PEGASUS and BART) and two fine-tuning datasets (XSum and CNN/DailyMail) demonstrate that our method enhances factual adaptiveness while achieving factual consistency on original datasets on par with the contrastive learning baseline.