Round-Trip Translation Reveals What Frontier Multilingual Benchmarks Miss
For researchers and developers of multilingual AI models, this work provides a more accurate and practical evaluation method that reveals gaps in real-world multilingual generation capabilities.
The paper shows that current multilingual benchmarks primarily measure mathematical reasoning and factual recall rather than multilingual proficiency, and proposes round-trip translation as a better evaluation method. Their benchmark, Lost in Translation (LiT), correlates almost perfectly (ρ = 0.94) with user ratings on LMArena.
Multilingual benchmarks guide the development of frontier models. Yet multilingual evaluations reported by frontier models are structured similar to popular reasoning and knowledge benchmarks, but across many languages. We show such benchmarks, and consequently multilingual evaluations, measure mathematical reasoning and factual recall, not multilingual proficiency. For example, thinking variants dramatically outperform instruct variants on these benchmarks, yet often perform worse on real-world multilingual tasks, such as LMArena. We propose a simple alternative: evaluate multilingual capability via round-trip translation. Given text in a source language, translate it to a target language and back; semantic gaps between the original and result expose failures in multilingual generation capabilities. Round-trip translation correlates almost perfectly (\r{ho} = 0.94) with user ratings on LMArena with our benchmark, requires no human reference translations, and does not require a more capable multilingual judge than tested models. Lastly, we introduce Lost in Translation (LiT), a challenging round-trip translation benchmark spanning widely spoken languages worldwide, for realistic evaluation of multilingual frontier models.