BookAsSumQA: An Evaluation Framework for Aspect-Based Book Summarization via Question Answering
This work addresses the problem of evaluating personalized summaries for long texts like books, which is incremental as it adapts existing QA-based evaluation to a new domain.
The paper tackled the challenge of evaluating aspect-based summarization for books by proposing BookAsSumQA, a QA-based framework that uses a narrative knowledge graph to generate aspect-specific questions, and found that RAG-based methods outperform LLM-based approaches on longer texts, with RAG showing higher efficiency as document length increases.
Aspect-based summarization aims to generate summaries that highlight specific aspects of a text, enabling more personalized and targeted summaries. However, its application to books remains unexplored due to the difficulty of constructing reference summaries for long text. To address this challenge, we propose BookAsSumQA, a QA-based evaluation framework for aspect-based book summarization. BookAsSumQA automatically generates aspect-specific QA pairs from a narrative knowledge graph to evaluate summary quality based on its question-answering performance. Our experiments using BookAsSumQA revealed that while LLM-based approaches showed higher accuracy on shorter texts, RAG-based methods become more effective as document length increases, making them more efficient and practical for aspect-based book summarization.