Learning to Revise References for Faithful Summarization
This addresses the issue of hallucination in summarization models for domains like healthcare, where data is limited and noisy, offering a practical solution without data filtering.
The paper tackled the problem of noisy reference summaries in summarization tasks by proposing a method to selectively rewrite unsupported sentences to align with source data, resulting in models trained on revised clinical references that were much more faithful, informative, and fluent according to metrics and human evaluation.
In real-world scenarios with naturally occurring datasets, reference summaries are noisy and may contain information that cannot be inferred from the source text. On large news corpora, removing low quality samples has been shown to reduce model hallucinations. Yet, for smaller, and/or noisier corpora, filtering is detrimental to performance. To improve reference quality while retaining all data, we propose a new approach: to selectively re-write unsupported reference sentences to better reflect source data. We automatically generate a synthetic dataset of positive and negative revisions by corrupting supported sentences and learn to revise reference sentences with contrastive learning. The intensity of revisions is treated as a controllable attribute so that, at inference, diverse candidates can be over-generated-then-rescored to balance faithfulness and abstraction. To test our methods, we extract noisy references from publicly available MIMIC-III discharge summaries for the task of hospital-course summarization, and vary the data on which models are trained. According to metrics and human evaluation, models trained on revised clinical references are much more faithful, informative, and fluent than models trained on original or filtered data.