With a Little Help from the Authors: Reproducing Human Evaluation of an MT Error Detector
This work addresses reproducibility issues in machine translation evaluation, highlighting challenges in experimental setup and human annotation variability, which is incremental as it builds on an existing study.
The authors attempted to reproduce a human evaluation experiment from a prior study on detecting over- and undertranslations in machine translation outputs, finding that while results generally confirmed the original conclusions, some statistically significant differences indicated high variability in human annotation.
This work presents our efforts to reproduce the results of the human evaluation experiment presented in the paper of Vamvas and Sennrich (2022), which evaluated an automatic system detecting over- and undertranslations (translations containing more or less information than the original) in machine translation (MT) outputs. Despite the high quality of the documentation and code provided by the authors, we discuss some problems we found in reproducing the exact experimental setup and offer recommendations for improving reproducibility. Our replicated results generally confirm the conclusions of the original study, but in some cases, statistically significant differences were observed, suggesting a high variability of human annotation.