NativQA: Multilingual Culturally-Aligned Natural Query for LLMs
This addresses the problem of inadequate benchmarking for regional and cultural specificities in LLMs, though it is incremental as it builds on existing QA dataset efforts.
The authors tackled the lack of region-specific, culturally-aligned natural question answering datasets for evaluating large language models by proposing NativQA, a scalable framework, and created MultiNativQA with ~64k manually annotated QA pairs in seven languages, which they used to benchmark various LLMs.
Natural Question Answering (QA) datasets play a crucial role in evaluating the capabilities of large language models (LLMs), ensuring their effectiveness in real-world applications. Despite the numerous QA datasets that have been developed and some work has been done in parallel, there is a notable lack of a framework and large scale region-specific datasets queried by native users in their own languages. This gap hinders the effective benchmarking and the development of fine-tuned models for regional and cultural specificities. In this study, we propose a scalable, language-independent framework, NativQA, to seamlessly construct culturally and regionally aligned QA datasets in native languages, for LLM evaluation and tuning. We demonstrate the efficacy of the proposed framework by designing a multilingual natural QA dataset, MultiNativQA, consisting of ~64k manually annotated QA pairs in seven languages, ranging from high to extremely low resource, based on queries from native speakers from 9 regions covering 18 topics. We benchmark open- and closed-source LLMs with the MultiNativQA dataset. We made the MultiNativQA dataset(https://huggingface.co/datasets/QCRI/MultiNativQA), and other experimental scripts(https://gitlab.com/nativqa/multinativqa) publicly available for the community.