Meta-Evaluation of Translation Evaluation Methods: a systematic up-to-date overview
It offers a comprehensive meta-evaluation for researchers and practitioners in machine translation to understand and improve evaluation practices, but it is incremental as it synthesizes existing work.
This tutorial provides a systematic overview of translation evaluation methods, covering traditional human judgments, automatic metrics, and recent advancements using pre-trained language models for metric customization, while also addressing statistical confidence estimation for human evaluation.
Starting from the 1950s, Machine Translation (MT) was challenged by different scientific solutions, which included rule-based methods, example-based and statistical models (SMT), to hybrid models, and very recent years the neural models (NMT). While NMT has achieved a huge quality improvement in comparison to conventional methodologies, by taking advantage of a huge amount of parallel corpora available from the internet and the recently developed super computational power support with an acceptable cost, it struggles to achieve real human parity in many domains and most language pairs, if not all of them. Alongside the long road of MT research and development, quality evaluation metrics played very important roles in MT advancement and evolution. In this tutorial, we overview the traditional human judgement criteria, automatic evaluation metrics, unsupervised quality estimation models, as well as the meta-evaluation of the evaluation methods. Among these, we will also cover the very recent work in the MT evaluation (MTE) fields, taking advantage of the large size of pre-trained language models for automatic metric customisation towards exactly deployed language pairs and domains. In addition, we also introduce the statistical confidence estimation regarding the sample size needed for human evaluation in real practice simulation. Full tutorial material is \textbf{available} to download at https://github.com/poethan/LREC22_MetaEval_Tutorial.