Charting the European LLM Benchmarking Landscape: A New Taxonomy and a Set of Best Practices
This work tackles the problem of fragmented and inadequate LLM benchmarking for European languages, which is incremental as it builds on existing benchmarking efforts.
The paper addresses the lack of systematic benchmarking for large language models (LLMs) in non-English languages by proposing a new taxonomy and best practices tailored to multilingual scenarios, aiming to improve coordination and cultural sensitivity in evaluations.
While new benchmarks for large language models (LLMs) are being developed continuously to catch up with the growing capabilities of new models and AI in general, using and evaluating LLMs in non-English languages remains a little-charted landscape. We give a concise overview of recent developments in LLM benchmarking, and then propose a new taxonomy for the categorization of benchmarks that is tailored to multilingual or non-English use scenarios. We further propose a set of best practices and quality standards that could lead to a more coordinated development of benchmarks for European languages. Among other recommendations, we advocate for a higher language and culture sensitivity of evaluation methods.