Calibrating Model-Based Evaluation Metrics for Summarization
For researchers and practitioners in summarization evaluation, this work addresses the miscalibration and resource requirements of model-based metrics, offering a practical calibration method.
The paper proposes a framework for generating calibrated summary evaluation scores without reference summaries or human annotations, using group isotonic regression binning (GIRB). Experiments on seven datasets show consistent improvements over baselines.
Recent advances in summary evaluation are based on model-based metrics to assess quality dimensions, such as completeness, conciseness, and faithfulness. However, these methods often require large language models, and predicted scores are frequently miscalibrated, limiting their reliability. Moreover, evaluating the average quality across different summaries for a single document typically requires access to multiple reference summaries. Here, we propose a general framework that generates individual and average proxy scores without relying on reference summaries, human annotations, or expensive model-based metrics. We also propose group isotonic regression binning (GIRB), a calibration method that adjusts the raw predictions to better align with ground-truth evaluation metrics. While we focus on continuous-value scenarios, such as summarization, the method is applicable to discrete-value tasks, such as question answering. Experiments on seven datasets demonstrate that our approach consistently outperforms existing baselines.