Faithfulness-Aware Decoding Strategies for Abstractive Summarization
This addresses the problem of improving faithfulness in abstractive summarization for NLP applications, though it is incremental as it builds on existing generation techniques.
The study investigated how decoding strategies like beam search and nucleus sampling affect faithfulness in abstractive summarization, finding that beam search with large beam sizes produces the most faithful summaries, and proposed two faithfulness-aware generation methods that significantly improved faithfulness across datasets as evaluated by metrics and human evaluation.
Despite significant progress in understanding and improving faithfulness in abstractive summarization, the question of how decoding strategies affect faithfulness is less studied. We present a systematic study of the effect of generation techniques such as beam search and nucleus sampling on faithfulness in abstractive summarization. We find a consistent trend where beam search with large beam sizes produces the most faithful summaries while nucleus sampling generates the least faithful ones. We propose two faithfulness-aware generation methods to further improve faithfulness over current generation techniques: (1) ranking candidates generated by beam search using automatic faithfulness metrics and (2) incorporating lookahead heuristics that produce a faithfulness score on the future summary. We show that both generation methods significantly improve faithfulness across two datasets as evaluated by four automatic faithfulness metrics and human evaluation. To reduce computational cost, we demonstrate a simple distillation approach that allows the model to generate faithful summaries with just greedy decoding. Our code is publicly available at https://github.com/amazon-science/faithful-summarization-generation