Improving Faithfulness of Large Language Models in Summarization via Sliding Generation and Self-Consistency
This addresses the issue of hallucinations in LLM-generated summaries, which is a critical problem for users relying on accurate information extraction, though it is an incremental improvement over existing methods.
The paper tackles the problem of factual inconsistency (hallucinations) in large language models during summarization by proposing SliSum, a method that uses sliding windows and self-consistency to generate more faithful summaries. The result shows that SliSum significantly improves faithfulness for models like LLaMA-2, Claude-2, and GPT-3.5 in both short and long text summarization without additional fine-tuning or resources.
Despite large language models (LLMs) have demonstrated impressive performance in various tasks, they are still suffering from the factual inconsistency problem called hallucinations. For instance, LLMs occasionally generate content that diverges from source article, and prefer to extract information that appears at the beginning and end of the context, especially in long document summarization. Inspired by these findings, we propose to improve the faithfulness of LLMs in summarization by impelling them to process the entire article more fairly and faithfully. We present a novel summary generation strategy, namely SliSum, which exploits the ideas of sliding windows and self-consistency. Specifically, SliSum divides the source article into overlapping windows, and utilizes LLM to generate local summaries for the content in the windows. Finally, SliSum aggregates all local summaries using clustering and majority voting algorithm to produce more faithful summary of entire article. Extensive experiments demonstrate that SliSum significantly improves the faithfulness of diverse LLMs including LLaMA-2, Claude-2 and GPT-3.5 in both short and long text summarization, while maintaining their fluency and informativeness and without additional fine-tuning and resources. We further conduct qualitative and quantitative studies to investigate why SliSum works and impacts of hyperparameters in SliSum on performance.