Measuring the Effect of Disfluency in Multilingual Knowledge Probing Benchmarks
This work addresses a methodological issue for researchers using multilingual benchmarks, offering a practical solution to improve interpretability, though it is incremental as it builds on existing datasets.
The study tackled the problem of ungrammatical prompts in multilingual knowledge probing benchmarks like MLAMA, which distort LLM performance scores, and found that using whole-sentence translations from Google Translate or ChatGPT significantly increased knowledge retrieval scores across sampled languages.
For multilingual factual knowledge assessment of LLMs, benchmarks such as MLAMA use template translations that do not take into account the grammatical and semantic information of the named entities inserted in the sentence. This leads to numerous instances of ungrammaticality or wrong wording of the final prompts, which complicates the interpretation of scores, especially for languages that have a rich morphological inventory. In this work, we sample 4 Slavic languages from the MLAMA dataset and compare the knowledge retrieval scores between the initial (templated) MLAMA dataset and its sentence-level translations made by Google Translate and ChatGPT. We observe a significant increase in knowledge retrieval scores, and provide a qualitative analysis for possible reasons behind it. We also make an additional analysis of 5 more languages from different families and see similar patterns. Therefore, we encourage the community to control the grammaticality of highly multilingual datasets for higher and more interpretable results, which is well approximated by whole sentence translation with neural MT or LLM systems. The dataset and all related code is published at the Github repository: https://github.com/ZurichNLP/Fluent-mLAMA.