Summarization is (Almost) Dead
This suggests that conventional summarization research may be obsolete in the LLM era, though incremental improvements in datasets and evaluation are still needed.
The paper investigates the zero-shot summarization capability of large language models (LLMs) across five tasks, finding that human evaluators prefer LLM-generated summaries over human-written and fine-tuned model summaries, with better factual consistency and fewer hallucinations.
How well can large language models (LLMs) generate summaries? We develop new datasets and conduct human evaluation experiments to evaluate the zero-shot generation capability of LLMs across five distinct summarization tasks. Our findings indicate a clear preference among human evaluators for LLM-generated summaries over human-written summaries and summaries generated by fine-tuned models. Specifically, LLM-generated summaries exhibit better factual consistency and fewer instances of extrinsic hallucinations. Due to the satisfactory performance of LLMs in summarization tasks (even surpassing the benchmark of reference summaries), we believe that most conventional works in the field of text summarization are no longer necessary in the era of LLMs. However, we recognize that there are still some directions worth exploring, such as the creation of novel datasets with higher quality and more reliable evaluation methods.