The price of debiasing automatic metrics in natural language evaluation
This work addresses the cost-efficiency challenge in evaluating NLP systems for researchers and practitioners, but it is incremental as it builds on existing methods with limited practical gains.
The paper tackled the problem of high cost in human evaluation for NLP systems by proposing an unbiased estimator that combines automatic metrics with human judgments using control variates, but achieved only a 7-13% cost reduction in summarization and QA tasks. It proved this estimator is optimal, identifying bottlenecks in automatic metrics and human prompts for further improvements.
For evaluating generation systems, automatic metrics such as BLEU cost nothing to run but have been shown to correlate poorly with human judgment, leading to systematic bias against certain model improvements. On the other hand, averaging human judgments, the unbiased gold standard, is often too expensive. In this paper, we use control variates to combine automatic metrics with human evaluation to obtain an unbiased estimator with lower cost than human evaluation alone. In practice, however, we obtain only a 7-13% cost reduction on evaluating summarization and open-response question answering systems. We then prove that our estimator is optimal: there is no unbiased estimator with lower cost. Our theory further highlights the two fundamental bottlenecks---the automatic metric and the prompt shown to human evaluators---both of which need to be improved to obtain greater cost savings.