Has Machine Translation Evaluation Achieved Human Parity? The Human Reference and the Limits of Progress
This work addresses the MT evaluation community's challenge in reliably measuring progress, highlighting incremental insights into metric performance bounds.
The study investigates whether machine translation evaluation metrics have reached human parity by comparing them to human baselines, finding that state-of-the-art metrics often perform on par with or better than humans, suggesting potential parity but with caution due to measurement limitations.
In Machine Translation (MT) evaluation, metric performance is assessed based on agreement with human judgments. In recent years, automatic metrics have demonstrated increasingly high levels of agreement with humans. To gain a clearer understanding of metric performance and establish an upper bound, we incorporate human baselines in the MT meta-evaluation, that is, the assessment of MT metrics' capabilities. Our results show that human annotators are not consistently superior to automatic metrics, with state-of-the-art metrics often ranking on par with or higher than human baselines. Despite these findings suggesting human parity, we discuss several reasons for caution. Finally, we explore the broader implications of our results for the research field, asking: Can we still reliably measure improvements in MT evaluation? With this work, we aim to shed light on the limits of our ability to measure progress in the field, fostering discussion on an issue that we believe is crucial to the entire MT evaluation community.