COMET: A Neural Framework for MT Evaluation
This provides a more accurate and robust evaluation method for machine translation systems, benefiting researchers and practitioners in natural language processing.
The authors tackled the problem of evaluating machine translation quality by developing COMET, a neural framework that trains multilingual evaluation models using cross-lingual pretrained language models, achieving new state-of-the-art correlation with human judgements on the WMT 2019 Metrics shared task.
We present COMET, a neural framework for training multilingual machine translation evaluation models which obtains new state-of-the-art levels of correlation with human judgements. Our framework leverages recent breakthroughs in cross-lingual pretrained language modeling resulting in highly multilingual and adaptable MT evaluation models that exploit information from both the source input and a target-language reference translation in order to more accurately predict MT quality. To showcase our framework, we train three models with different types of human judgements: Direct Assessments, Human-mediated Translation Edit Rate and Multidimensional Quality Metrics. Our models achieve new state-of-the-art performance on the WMT 2019 Metrics shared task and demonstrate robustness to high-performing systems.