A Measure of the System Dependence of Automated Metrics
This addresses a critical issue for researchers and practitioners in NLP who rely on automated metrics to assess translation quality, though it is incremental as it builds on existing correlation-based evaluations.
The paper tackles the problem of ensuring fairness and consistency in automated metrics for Machine Translation by introducing a method to evaluate how metrics treat different systems, moving beyond just correlation with human judgments.
Automated metrics for Machine Translation have made significant progress, with the goal of replacing expensive and time-consuming human evaluations. These metrics are typically assessed by their correlation with human judgments, which captures the monotonic relationship between human and metric scores. However, we argue that it is equally important to ensure that metrics treat all systems fairly and consistently. In this paper, we introduce a method to evaluate this aspect.