Diagnosing Translated Benchmarks: An Automated Quality Assurance Study of the EU20 Benchmark Suite
This addresses the issue of noise and uneven quality in translated benchmarks for researchers and practitioners, offering scalable tools to prioritize review, though it is incremental as it builds on existing translation methods.
The study tackled the problem of unreliable machine-translated benchmark datasets by developing an automated quality assurance approach to measure and verify translation reliability in the EU20 benchmark suite, resulting in cleaned datasets and code for reproducibility.
Machine-translated benchmark datasets reduce costs and offer scale, but noise, loss of structure, and uneven quality weaken confidence. What matters is not merely whether we can translate, but also whether we can measure and verify translation reliability at scale. We study translation quality in the EU20 benchmark suite, which comprises five established benchmarks translated into 20 languages, via a three-step automated quality assurance approach: (i) a structural corpus audit with targeted fixes; (ii) quality profiling using a neural metric (COMET, reference-free and reference-based) with translation service comparisons (DeepL / ChatGPT / Google); and (iii) an LLM-based span-level translation error landscape. Trends are consistent: datasets with lower COMET scores exhibit a higher share of accuracy/mistranslation errors at span level (notably HellaSwag; ARC is comparatively clean). Reference-based COMET on MMLU against human-edited samples points in the same direction. We release cleaned/corrected versions of the EU20 datasets, and code for reproducibility. In sum, automated quality assurance offers practical, scalable indicators that help prioritize review -- complementing, not replacing, human gold standards.