Towards Multilingual LLM Evaluation for European Languages
This work addresses the problem of multilingual LLM evaluation for researchers and practitioners in NLP, though it is incremental as it builds on existing benchmarks through translation.
The paper tackled the challenge of evaluating Large Language Models (LLMs) consistently across European languages by introducing a multilingual evaluation approach using translated benchmarks, assessing 40 LLMs across 21 languages and creating new datasets like EU20-MMLU and EU20-HellaSwag.
The rise of Large Language Models (LLMs) has revolutionized natural language processing across numerous languages and tasks. However, evaluating LLM performance in a consistent and meaningful way across multiple European languages remains challenging, especially due to the scarcity of language-parallel multilingual benchmarks. We introduce a multilingual evaluation approach tailored for European languages. We employ translated versions of five widely-used benchmarks to assess the capabilities of 40 LLMs across 21 European languages. Our contributions include examining the effectiveness of translated benchmarks, assessing the impact of different translation services, and offering a multilingual evaluation framework for LLMs that includes newly created datasets: EU20-MMLU, EU20-HellaSwag, EU20-ARC, EU20-TruthfulQA, and EU20-GSM8K. The benchmarks and results are made publicly available to encourage further research in multilingual LLM evaluation.