ECLeKTic: a Novel Challenge Set for Evaluation of Cross-Lingual Knowledge Transfer
This addresses the need for reliable evaluation methods for cross-lingual knowledge transfer in LLMs, which is crucial for achieving equitable performance across languages, though it is incremental as it focuses on dataset creation and benchmarking.
The authors tackled the problem of measuring cross-lingual knowledge transfer in LLMs by creating ECLeKTic, a multilingual QA dataset, and found that current SOTA models struggle to effectively share knowledge across languages, with performance gaps observed in evaluations of 8 models.
To achieve equitable performance across languages, large language models (LLMs) must be able to abstract knowledge beyond the language in which it was learnt. However, the current literature lacks reliable ways to measure LLMs' capability of such cross-lingual knowledge transfer. To that end, we present ECLeKTic, a multilingual closed-book QA dataset that Evaluates Cross-Lingual Knowledge Transfer in a simple, black-box manner. Concretely, we used the presence and absence of Wikipedia articles in 12 languages to detect pieces of information that were likely available during pre-training in one of the languages but not in the others. We curate ECLeKTic as a set of fact-seeking questions over this kind of information, in all the different languages. Therefore, in order to solve ECLeKTic the model is required to transfer knowledge between languages. We evaluated 8 LLMs and showed that current SOTA models struggle to effectively share knowledge across languages, even if they can predict the answer for questions in the language in which the knowledge was acquired.