Separating form and meaning: Using self-consistency to quantify task understanding across multiple senses
This addresses the problem of creating robust evaluations for LLMs, offering a scalable method for researchers and developers, though it is incremental in applying self-consistency to multilingual contexts.
The authors tackled the challenge of evaluating large language models' understanding by proposing a new paradigm that measures consistency across different senses of the same meaning, such as multiple languages, and found that ChatGPT's multilingual consistency is lacking, indicating its understanding is not language-independent.
At the staggering pace with which the capabilities of large language models (LLMs) are increasing, creating future-proof evaluation sets to assess their understanding becomes more and more challenging. In this paper, we propose a novel paradigm for evaluating LLMs which leverages the idea that correct world understanding should be consistent across different (Fregean) senses of the same meaning. Accordingly, we measure understanding not in terms of correctness but by evaluating consistency across multiple senses that are generated by the model itself. We showcase our approach by instantiating a test where the different senses are different languages, hence using multilingual self-consistency as a litmus test for the model's understanding and simultaneously addressing the important topic of multilinguality. Taking one of the latest versions of ChatGPT as our object of study, we evaluate multilingual consistency for two different tasks across three different languages. We show that its multilingual consistency is still lacking, and that its task and world understanding are thus not language-independent. As our approach does not require any static evaluation corpora in languages other than English, it can easily and cheaply be extended to different languages and tasks and could become an integral part of future benchmarking efforts.