GUMsley: Evaluating Entity Salience in Summarization for 12 English Genres
This addresses the need for better entity salience evaluation in NLP, particularly for information retrieval and controllable summarization, though it is incremental as it builds on existing datasets and methods.
The authors tackled the problem of identifying salient entities in summarization by introducing GUMsley, a dataset covering 12 English genres with annotations, and found that pre-trained SOTA models and zero-shot LLM prompting perform poorly in capturing these entities, while using salient entities improves summarization quality and reduces hallucinations.
As NLP models become increasingly capable of understanding documents in terms of coherent entities rather than strings, obtaining the most salient entities for each document is not only an important end task in itself but also vital for Information Retrieval (IR) and other downstream applications such as controllable summarization. In this paper, we present and evaluate GUMsley, the first entity salience dataset covering all named and non-named salient entities for 12 genres of English text, aligned with entity types, Wikification links and full coreference resolution annotations. We promote a strict definition of salience using human summaries and demonstrate high inter-annotator agreement for salience based on whether a source entity is mentioned in the summary. Our evaluation shows poor performance by pre-trained SOTA summarization models and zero-shot LLM prompting in capturing salient entities in generated summaries. We also show that predicting or providing salient entities to several model architectures enhances performance and helps derive higher-quality summaries by alleviating the entity hallucination problem in existing abstractive summarization.