Expanding the WMT24++ Benchmark with Rumantsch Grischun, Sursilvan, Sutsilvan, Surmiran, Puter, and Vallader
This work addresses the problem of limited evaluation data for low-resource languages like Romansh, benefiting NLP researchers and communities, though it is incremental as it extends an existing benchmark.
The paper tackles the lack of machine translation evaluation resources for Romansh by creating a benchmark for six of its varieties, integrated with the WMT24++ framework, and finds that translation into Romansh remains challenging while out-of-Romansh translation performs relatively well.
The Romansh language, spoken in Switzerland, has limited resources for machine translation evaluation. In this paper, we present a benchmark for six varieties of Romansh: Rumantsch Grischun, a supra-regional variety, and five regional varieties: Sursilvan, Sutsilvan, Surmiran, Puter, and Vallader. Our reference translations were created by human translators based on the WMT24++ benchmark, which ensures parallelism with more than 55 other languages. An automatic evaluation of existing MT systems and LLMs shows that translation out of Romansh into German is handled relatively well for all the varieties, but translation into Romansh is still challenging.