Tackling Query-Focused Summarization as A Knowledge-Intensive Task: A Pilot Study
This addresses the problem of making query-focused summarization more realistic for scenarios where relevant documents are not readily available, though it is incremental as it adapts existing datasets and benchmarks.
The authors tackled query-focused summarization without pre-annotated relevant documents by reframing it as a knowledge-intensive task requiring retrieval from a corpus, and they found that state-of-the-art models performed significantly worse in this new setting, indicating it is more challenging.
Query-focused summarization (QFS) requires generating a summary given a query using a set of relevant documents. However, such relevant documents should be annotated manually and thus are not readily available in realistic scenarios. To address this limitation, we tackle the QFS task as a knowledge-intensive (KI) task without access to any relevant documents. Instead, we assume that these documents are present in a large-scale knowledge corpus and should be retrieved first. To explore this new setting, we build a new dataset (KI-QFS) by adapting existing QFS datasets. In this dataset, answering the query requires document retrieval from a knowledge corpus. We construct three different knowledge corpora, and we further provide relevance annotations to enable retrieval evaluation. Finally, we benchmark the dataset with state-of-the-art QFS models and retrieval-enhanced models. The experimental results demonstrate that QFS models perform significantly worse on KI-QFS compared to the original QFS task, indicating that the knowledge-intensive setting is much more challenging and offers substantial room for improvement. We believe that our investigation will inspire further research into addressing QFS in more realistic scenarios.