The Inside Story: Towards Better Understanding of Machine Translation Neural Evaluation Metrics
This work addresses the interpretability problem for researchers and practitioners using neural evaluation metrics in machine translation, though it is incremental as it builds on existing metrics.
The authors tackled the lack of transparency in neural machine translation evaluation metrics by developing and comparing explainability methods, revealing that these metrics effectively leverage token-level information to identify translation errors, as validated through comparisons with human annotations and synthetic errors.
Neural metrics for machine translation evaluation, such as COMET, exhibit significant improvements in their correlation with human judgments, as compared to traditional metrics based on lexical overlap, such as BLEU. Yet, neural metrics are, to a great extent, "black boxes" returning a single sentence-level score without transparency about the decision-making process. In this work, we develop and compare several neural explainability methods and demonstrate their effectiveness for interpreting state-of-the-art fine-tuned neural metrics. Our study reveals that these metrics leverage token-level information that can be directly attributed to translation errors, as assessed through comparison of token-level neural saliency maps with Multidimensional Quality Metrics (MQM) annotations and with synthetically-generated critical translation errors. To ease future research, we release our code at: https://github.com/Unbabel/COMET/tree/explainable-metrics.