DomainSum: A Hierarchical Benchmark for Fine-Grained Domain Shift in Abstractive Text Summarization
This addresses the issue of domain generalization in summarization for researchers, but it is incremental as it focuses on benchmarking rather than novel solutions.
The paper tackles the problem of domain shifts in abstractive text summarization by introducing DomainSum, a hierarchical benchmark that categorizes shifts into genre, style, and topic levels, and evaluates pre-trained and large language models, showing performance variations in cross-domain settings.
Most research on abstractive summarization focuses on single-domain applications, often neglecting how domain shifts between documents affect performance and the generalization ability of summarization models. To address this issue, we introduce DomainSum, a hierarchical benchmark designed to capture fine-grained domain shifts in abstractive summarization. We categorize these shifts into three levels: genre, style, and topic, and demonstrate through comprehensive benchmark analysis that they follow a hierarchical structure. Furthermore, we evaluate the domain generalization capabilities of commonly used pre-trained language models (PLMs) and large language models (LLMs) in in-domain and cross-domain settings.