Danoliteracy of Generative Large Language Models
This work addresses the problem of evaluating GLLMs in low-resource languages for researchers and practitioners, but it is incremental as it focuses on creating a specific benchmark rather than a new method.
The authors tackled the lack of evaluation benchmarks for Generative Large Language Models (GLLMs) in low-resource languages like Danish by creating a benchmark called Danoliteracy to assess Danish language and cultural competency across eight scenarios, finding that GPT-4 and Claude Opus models performed best with a correlation to human feedback of ρ ~ 0.8 and identifying a strong underlying factor explaining 95% of performance variance.
The language technology moonshot moment of Generative Large Language Models (GLLMs) was not limited to English: These models brought a surge of technological applications, investments, and hype to low-resource languages as well. However, the capabilities of these models in languages such as Danish were, until recently, difficult to verify beyond qualitative demonstrations due to a lack of applicable evaluation corpora. We present a GLLM benchmark to evaluate \emph{Danoliteracy}, a measure of Danish language and cultural competency across eight diverse scenarios such as Danish citizenship tests and abstractive social media question answering. This limited-size benchmark was found to produce a robust ranking that correlates to human feedback at $ρ\sim 0.8$ with GPT-4 and Claude Opus models achieving the highest rankings. Analyzing these model results across scenarios, we find one strong underlying factor explaining $95\%$ of scenario performance variance for GLLMs in Danish, suggesting a $g$ factor of model consistency in language adaptation.