Optimizing the Factual Correctness of a Summary: A Study of Summarizing Radiology Reports
This work addresses the critical need for factual correctness in summarization for real-world applications like radiology, where errors can have serious consequences, representing an incremental improvement over existing neural methods.
The authors tackled the problem of factual correctness in neural abstractive summarization models, particularly for radiology reports, by developing a framework that automatically evaluates and optimizes summaries for factual accuracy using reinforcement learning, resulting in substantial improvements in factual correctness and overall quality that approach human-authored summaries.
Neural abstractive summarization models are able to generate summaries which have high overlap with human references. However, existing models are not optimized for factual correctness, a critical metric in real-world applications. In this work, we develop a general framework where we evaluate the factual correctness of a generated summary by fact-checking it automatically against its reference using an information extraction module. We further propose a training strategy which optimizes a neural summarization model with a factual correctness reward via reinforcement learning. We apply the proposed method to the summarization of radiology reports, where factual correctness is a key requirement. On two separate datasets collected from hospitals, we show via both automatic and human evaluation that the proposed approach substantially improves the factual correctness and overall quality of outputs over a competitive neural summarization system, producing radiology summaries that approach the quality of human-authored ones.