Common to Whom? Regional Cultural Commonsense and LLM Bias in India
This addresses the issue of cultural bias in LLMs for users in culturally heterogeneous nations like India, providing a generalizable framework for evaluation, though it is incremental in extending existing benchmarks to regional levels.
The paper tackled the problem of assuming uniform cultural commonsense within nations by introducing Indica, a benchmark for India, and found that only 39.4% of questions had agreement across five regions, showing cultural commonsense is predominantly regional. It evaluated eight LLMs, revealing low accuracy (13.4%-20.9%) on region-specific questions and geographic bias, with models over-selecting Central and North India by 30-40%.
Existing cultural commonsense benchmarks treat nations as monolithic, assuming uniform practices within national boundaries. But does cultural commonsense hold uniformly within a nation, or does it vary at the sub-national level? We introduce Indica, the first benchmark designed to test LLMs' ability to address this question, focusing on India - a nation of 28 states, 8 union territories, and 22 official languages. We collect human-annotated answers from five Indian regions (North, South, East, West, and Central) across 515 questions spanning 8 domains of everyday life, yielding 1,630 region-specific question-answer pairs. Strikingly, only 39.4% of questions elicit agreement across all five regions, demonstrating that cultural commonsense in India is predominantly regional, not national. We evaluate eight state-of-the-art LLMs and find two critical gaps: models achieve only 13.4%-20.9% accuracy on region-specific questions, and they exhibit geographic bias, over-selecting Central and North India as the "default" (selected 30-40% more often than expected) while under-representing East and West. Beyond India, our methodology provides a generalizable framework for evaluating cultural commonsense in any culturally heterogeneous nation, from question design grounded in anthropological taxonomy, to regional data collection, to bias measurement.