ACES: Translation Accuracy Challenge Sets for Evaluating Machine Translation Metrics
This work addresses the need for better segment-level evaluation of machine translation metrics, particularly for accuracy errors in critical domains like legal and medical, but it is incremental as it builds on existing evaluation frameworks.
The authors tackled the problem of evaluating machine translation metrics by curating ACES, a challenge set with 68 accuracy error phenomena, and used it to analyze a wide range of metrics, leading to recommendations such as combining metrics and focusing on source-based modeling.
As machine translation (MT) metrics improve their correlation with human judgement every year, it is crucial to understand the limitations of such metrics at the segment level. Specifically, it is important to investigate metric behaviour when facing accuracy errors in MT because these can have dangerous consequences in certain contexts (e.g., legal, medical). We curate ACES, a translation accuracy challenge set, consisting of 68 phenomena ranging from simple perturbations at the word/character level to more complex errors based on discourse and real-world knowledge. We use ACES to evaluate a wide range of MT metrics including the submissions to the WMT 2022 metrics shared task and perform several analyses leading to general recommendations for metric developers. We recommend: a) combining metrics with different strengths, b) developing metrics that give more weight to the source and less to surface-level overlap with the reference and c) explicitly modelling additional language-specific information beyond what is available via multilingual embeddings.