Rethinking Scientific Summarization Evaluation: Grounding Explainable Metrics on Facet-aware Benchmark
This work addresses the need for better evaluation methods in scientific summarization, which is crucial for researchers and practitioners dealing with complex scientific texts, though it is incremental as it builds on existing LLM-based approaches.
The paper tackled the problem of evaluating scientific summarization by highlighting inadequacies of traditional metrics and introduced a Facet-aware Metric (FM) with a new dataset, finding that FM provides a more logical evaluation approach and that fine-tuned smaller models can compete with large language models in this domain.
The summarization capabilities of pretrained and large language models (LLMs) have been widely validated in general areas, but their use in scientific corpus, which involves complex sentences and specialized knowledge, has been less assessed. This paper presents conceptual and experimental analyses of scientific summarization, highlighting the inadequacies of traditional evaluation methods, such as $n$-gram, embedding comparison, and QA, particularly in providing explanations, grasping scientific concepts, or identifying key content. Subsequently, we introduce the Facet-aware Metric (FM), employing LLMs for advanced semantic matching to evaluate summaries based on different aspects. This facet-aware approach offers a thorough evaluation of abstracts by decomposing the evaluation task into simpler subtasks.Recognizing the absence of an evaluation benchmark in this domain, we curate a Facet-based scientific summarization Dataset (FD) with facet-level annotations. Our findings confirm that FM offers a more logical approach to evaluating scientific summaries. In addition, fine-tuned smaller models can compete with LLMs in scientific contexts, while LLMs have limitations in learning from in-context information in scientific domains. This suggests an area for future enhancement of LLMs.