ODSum: New Benchmarks for Open Domain Multi-Document Summarization
This work addresses the need for better benchmarks in open-domain multi-document summarization, though it is incremental as it builds on existing datasets and methods.
The paper tackles the problem of open-domain multi-document summarization by introducing ODSum, a new benchmark dataset with interdependent documents, and finds that large language models suffer significant performance loss due to retrieval errors, with experiments showing variances in evaluation metrics.
Open-domain Multi-Document Summarization (ODMDS) is a critical tool for condensing vast arrays of documents into coherent, concise summaries. With a more inter-related document set, there does not necessarily exist a correct answer for the retrieval, making it hard to measure the retrieving performance. We propose a rule-based method to process query-based document summarization datasets into ODMDS datasets. Based on this method, we introduce a novel dataset, ODSum, a sophisticated case with its document index interdependent and often interrelated. We tackle ODMDS with the \textit{retrieve-then-summarize} method, and the performance of a list of retrievers and summarizers is investigated. Through extensive experiments, we identify variances in evaluation metrics and provide insights into their reliability. We also found that LLMs suffer great performance loss from retrieving errors. We further experimented methods to improve the performance as well as investigate their robustness against imperfect retrieval. We will release our data and code at https://github.com/yale-nlp/ODSum.