SEval-Ex: A Statement-Level Framework for Explainable Summarization Evaluation
This addresses the problem of interpretable summarization evaluation for NLP researchers and practitioners, offering a novel method that balances performance and explainability.
The paper tackles the challenge of evaluating text summarization quality by introducing SEval-Ex, a framework that decomposes evaluation into atomic statements to achieve both high performance and explainability, resulting in a state-of-the-art correlation of 0.580 on consistency with human judgments, surpassing GPT-4 based evaluators.
Evaluating text summarization quality remains a critical challenge in Natural Language Processing. Current approaches face a trade-off between performance and interpretability. We present SEval-Ex, a framework that bridges this gap by decomposing summarization evaluation into atomic statements, enabling both high performance and explainability. SEval-Ex employs a two-stage pipeline: first extracting atomic statements from text source and summary using LLM, then a matching between generated statements. Unlike existing approaches that provide only summary-level scores, our method generates detailed evidence for its decisions through statement-level alignments. Experiments on the SummEval benchmark demonstrate that SEval-Ex achieves state-of-the-art performance with 0.580 correlation on consistency with human consistency judgments, surpassing GPT-4 based evaluators (0.521) while maintaining interpretability. Finally, our framework shows robustness against hallucination.