LAG-MMLU: Benchmarking Frontier LLM Understanding in Latvian and Giriama
This addresses the lack of robust evaluation frameworks for low-resource languages, providing localized benchmarks for AI cultural contextualization, though it is incremental as it extends existing methods to new languages.
The study evaluated eight state-of-the-art LLMs on Latvian and Giriama using a curated MMLU subset, finding that OpenAI's o1 model scored 92.8% in English, 88.8% in Latvian, and 70.8% in Giriama, while others like Mistral-large and Llama-70B IT performed poorly.
As large language models (LLMs) rapidly advance, evaluating their performance is critical. LLMs are trained on multilingual data, but their reasoning abilities are mainly evaluated using English datasets. Hence, robust evaluation frameworks are needed using high-quality non-English datasets, especially low-resource languages (LRLs). This study evaluates eight state-of-the-art (SOTA) LLMs on Latvian and Giriama using a Massive Multitask Language Understanding (MMLU) subset curated with native speakers for linguistic and cultural relevance. Giriama is benchmarked for the first time. Our evaluation shows that OpenAI's o1 model outperforms others across all languages, scoring 92.8% in English, 88.8% in Latvian, and 70.8% in Giriama on 0-shot tasks. Mistral-large (35.6%) and Llama-70B IT (41%) have weak performance, on both Latvian and Giriama. Our results underscore the need for localized benchmarks and human evaluations in advancing cultural AI contextualization.