XLGoBench: Detecting cross-lingual skill gaps with algorithmic tasks
This work provides a new method for researchers to objectively quantify and identify cross-lingual performance disparities in large language models.
This paper introduces XLGoBench, a benchmark of synthetic algorithmic tasks designed to identify cross-lingual skill gaps in large language models. The benchmark reveals persistent cross-lingual gaps across multiple state-of-the-art models.
We introduce a set of synthetic algorithmic tasks to detect cross-lingual gaps in the abilities of large language models. Our benchmark is commensurate across languages, since it requires models to perform the same underlying task in different languages; scalable, since each task can be generated at varying levels of complexity allowing it to be adapted to models with different capabilities; quantifiable, since every task admits an objective notion of correctness; and transparent, since tasks are generated from simple templates that can be readily audited for translation errors. Because our benchmark focuses on algorithmic tasks, differential performance is a sufficient -- but not necessary -- indicator of cross-lingual gaps. Nevertheless, we show through extensive experiments that our benchmark exposes persistent cross-lingual gaps in multiple state-of-the-art models.