Can We Trust the Performance Evaluation of Uncertainty Estimation Methods in Text Summarization?
This work addresses the critical need for trustworthy uncertainty estimation in text summarization for risk-critical applications, but it is incremental as it focuses on benchmarking and evaluation rather than proposing new methods.
The paper tackles the problem of unreliable evaluation of uncertainty estimation methods in text summarization by introducing a comprehensive benchmark with 31 NLG metrics across four dimensions, evaluating two large language models and one pre-trained model on three datasets, and assessing 14 uncertainty estimation methods, emphasizing the need for multiple uncorrelated metrics and diverse methods for reliable evaluation.
Text summarization, a key natural language generation (NLG) task, is vital in various domains. However, the high cost of inaccurate summaries in risk-critical applications, particularly those involving human-in-the-loop decision-making, raises concerns about the reliability of uncertainty estimation on text summarization (UE-TS) evaluation methods. This concern stems from the dependency of uncertainty model metrics on diverse and potentially conflicting NLG metrics. To address this issue, we introduce a comprehensive UE-TS benchmark incorporating 31 NLG metrics across four dimensions. The benchmark evaluates the uncertainty estimation capabilities of two large language models and one pre-trained language model on three datasets, with human-annotation analysis incorporated where applicable. We also assess the performance of 14 common uncertainty estimation methods within this benchmark. Our findings emphasize the importance of considering multiple uncorrelated NLG metrics and diverse uncertainty estimation methods to ensure reliable and efficient evaluation of UE-TS techniques. Our code and data are available https://github.com/he159ok/Benchmark-of-Uncertainty-Estimation-Methods-in-Text-Summarization.