SANSKRITI: A Comprehensive Benchmark for Evaluating Language Models' Knowledge of Indian Culture
This addresses the need for culturally aware language models, particularly for Indian contexts, but is incremental as it focuses on creating a new benchmark rather than a novel method.
The authors tackled the problem of language models lacking understanding of local socio-cultural contexts by introducing SANSKRITI, a benchmark with 21,853 question-answer pairs covering Indian culture, and found significant disparities in model performance, with many struggling in region-specific contexts.
Language Models (LMs) are indispensable tools shaping modern workflows, but their global effectiveness depends on understanding local socio-cultural contexts. To address this, we introduce SANSKRITI, a benchmark designed to evaluate language models' comprehension of India's rich cultural diversity. Comprising 21,853 meticulously curated question-answer pairs spanning 28 states and 8 union territories, SANSKRITI is the largest dataset for testing Indian cultural knowledge. It covers sixteen key attributes of Indian culture: rituals and ceremonies, history, tourism, cuisine, dance and music, costume, language, art, festivals, religion, medicine, transport, sports, nightlife, and personalities, providing a comprehensive representation of India's cultural tapestry. We evaluate SANSKRITI on leading Large Language Models (LLMs), Indic Language Models (ILMs), and Small Language Models (SLMs), revealing significant disparities in their ability to handle culturally nuanced queries, with many models struggling in region-specific contexts. By offering an extensive, culturally rich, and diverse dataset, SANSKRITI sets a new standard for assessing and improving the cultural understanding of LMs.