CUS-QA: Local-Knowledge-Oriented Open-Ended Question Answering Dataset
This provides a benchmark for evaluating LLMs on regional knowledge tasks, but it is incremental as it builds on existing QA datasets with a new focus.
The authors introduced CUS-QA, a dataset for open-ended regional question answering with textual and visual modalities, and found that state-of-the-art LLMs achieved only about 50% accuracy on textual questions and below 30% on visual questions.
We introduce CUS-QA, a benchmark for open-ended regional question answering that encompasses both textual and visual modalities. We also provide strong baselines using state-of-the-art large language models (LLMs). Our dataset consists of manually curated questions and answers grounded in Wikipedia, created by native speakers from Czechia, Slovakia, and Ukraine, with accompanying English translations. It includes both purely textual questions and those requiring visual understanding. We evaluate state-of-the-art LLMs through prompting and complement this with human judgments of answer correctness. Using these human evaluations, we analyze the reliability of existing automatic evaluation metrics. Our baseline results show that even the best open-weight LLMs achieve only around 50% accuracy on textual questions and below 30% on visual questions. LLM-based evaluation metrics show strong correlation with human judgment, while traditional string-overlap metrics perform surprisingly well due to the prevalence of named entities in answers.