Enhancing Factuality through Consensus and Consistency in Summarization Using Minimum Bayes Risk Decoding
For researchers and practitioners in text summarization, this work offers a method to enhance summary factuality by combining consensus and consistency, though it is an incremental improvement over existing reranking approaches.
The paper addresses the challenge of improving factuality in summarization by proposing ConSUM, which reranks candidate summaries using both consensus among candidates (via Minimum Bayes Risk decoding) and consistency to the source document. The system achieves competitive performance with existing methods and is preferred in human evaluations.
Improving the quality of model-generated summaries, especially factuality, the accuracy of a summary with respect to its source content, remains a challenge. While reranking could select the optimal output from multiple generated candidates, it is limited to only using the source as guidance, resulting in unreliable summaries. To address this limitation, we propose ConSUM that reranks candidate summaries by considering two factors: consistency to the source document and consensus among the other candidates. Consensus is established using Minimum Bayes Risk (MBR) decoding over the set of generated summaries, while ensuring consistency by employing factuality-aware metrics that compare the summary against the source. Rigorous testing demonstrates that our system is competitive with existing methods, with human evaluations further confirming that its generated summaries are preferred over those from other systems. Our code is available at https://github.com/naist-nlp/ConSUM .