SummIt: Iterative Text Summarization via ChatGPT
This work addresses the issue of unreliable summaries for users needing accurate and tailored information, but it is incremental as it builds on existing large language models.
The paper tackles the problem of one-shot text summarization leading to hallucinations or missing details by proposing SummIt, an iterative framework using ChatGPT for self-evaluation and feedback, which was evaluated on three benchmark datasets and through human assessment.
Text summarization systems have made significant progress in recent years, but typically generate summaries in one single step. However, the one-shot summarization setting is sometimes inadequate, as the generated summary may contain hallucinations or overlook essential details related to the reader's interests. This paper addresses this limitation by proposing SummIt, an iterative text summarization framework based on large language models like ChatGPT. Our framework enables the model to refine the generated summary iteratively through self-evaluation and feedback, resembling humans' iterative process when drafting and revising summaries. Furthermore, we explore the potential benefits of integrating knowledge and topic extractors into the framework to enhance summary faithfulness and controllability. We automatically evaluate the performance of our framework on three benchmark summarization datasets. We also conduct a human evaluation to validate the effectiveness of the iterative refinements and identify a potential issue of over-correction.