Estonian WinoGrande Dataset: Comparative Analysis of LLM Performance on Human and Machine Translation
This work addresses the need for culturally adapted benchmarks to reliably evaluate LLMs in non-English languages, though it is incremental as it focuses on a specific dataset and language.
The researchers created an Estonian version of the WinoGrande benchmark through human translation and evaluated LLMs, finding slightly lower performance on the human-translated dataset compared to English and notably worse results on machine-translated data, with prompt engineering providing limited improvements.
In this paper, we present a localized and culturally adapted Estonian translation of the test set from the widely used commonsense reasoning benchmark, WinoGrande. We detail the translation and adaptation process carried out by translation specialists and evaluate the performance of both proprietary and open source models on the human translated benchmark. Additionally, we explore the feasibility of achieving high-quality machine translation by incorporating insights from the manual translation process into the design of a detailed prompt. This prompt is specifically tailored to address both the linguistic characteristics of Estonian and the unique translation challenges posed by the WinoGrande dataset. Our findings show that model performance on the human translated Estonian dataset is slightly lower than on the original English test set, while performance on machine-translated data is notably worse. Additionally, our experiments indicate that prompt engineering offers limited improvement in translation quality or model accuracy, and highlight the importance of involving language specialists in dataset translation and adaptation to ensure reliable and interpretable evaluations of language competency and reasoning in large language models.