MQM Re-Annotation: A Technique for Collaborative Evaluation of Machine Translation
This addresses evaluation reliability for machine translation researchers, but is an incremental improvement to the existing MQM paradigm.
The paper tackles the problem of evaluation noise in machine translation as models improve, by proposing MQM re-annotation where annotators review and edit existing annotations. The result shows that re-annotation yields higher-quality annotations, primarily by catching missed errors.
Human evaluation of machine translation is in an arms race with translation model quality: as our models get better, our evaluation methods need to be improved to ensure that quality gains are not lost in evaluation noise. To this end, we experiment with a two-stage version of the current state-of-the-art translation evaluation paradigm (MQM), which we call MQM re-annotation. In this setup, an MQM annotator reviews and edits a set of pre-existing MQM annotations, that may have come from themselves, another human annotator, or an automatic MQM annotation system. We demonstrate that rater behavior in re-annotation aligns with our goals, and that re-annotation results in higher-quality annotations, mostly due to finding errors that were missed during the first pass.