Span-Level Machine Translation Meta-Evaluation
This addresses a methodological gap for researchers and practitioners in machine translation evaluation, but it is incremental as it builds on existing error detection techniques.
The paper tackled the problem of reliably measuring the capabilities of automatic machine translation evaluators that detect errors, by investigating different span-level metrics and showing they yield inconsistent rankings. They proposed a new meta-evaluation strategy called MPP, which they used to assess state-of-the-art error detection methods.
Machine Translation (MT) and automatic MT evaluation have improved dramatically in recent years, enabling numerous novel applications. Automatic evaluation techniques have evolved from producing scalar quality scores to precisely locating translation errors and assigning them error categories and severity levels. However, it remains unclear how to reliably measure the evaluation capabilities of auto-evaluators that do error detection, as no established technique exists in the literature. This work investigates different implementations of span-level precision, recall, and F-score, showing that seemingly similar approaches can yield substantially different rankings, and that certain widely-used techniques are unsuitable for evaluating MT error detection. We propose "match with partial overlap and partial credit" (MPP) with micro-averaging as a robust meta-evaluation strategy and release code for its use publicly. Finally, we use MPP to assess the state of the art in MT error detection.