Quantifying the Impact of Translation Errors on Multilingual LLM Evaluation
For researchers using translated benchmarks to evaluate multilingual LLMs, this work quantifies the reliability threat posed by translation errors, showing they are a significant confound beyond source-side issues.
The study reveals that translation errors in machine-translated benchmarks cause measurable accuracy drops in multilingual LLM evaluation, with span-level agreement between automatic error detectors and human annotations remaining non-trivial.
Machine-translated benchmarks are widely used to assess the multilingual capabilities of large language models (LLMs), yet translation errors in these benchmarks remain underexplored, raising concerns about the reliability and comparability of multilingual evaluation. We address two practical gaps: (i) how well automatic MQM-style error spans from LLM judges and a span-aware QE baseline (xCOMET-XXL) match expert human span annotations on benchmark translations, and (ii) how strongly translation errors (as opposed to source-side issues in the English original) explain accuracy drops on translated benchmarks. We find that span agreement is non-trivial on naturally occurring benchmark translations, and that target-side translation errors are consistently associated with measurable, percentage-point drops in translated accuracy even after controlling for English correctness and source-side anomalies.