AmchiBias: Measuring Stereotypical Bias in Goan Identity Groups with a Minimal Pair Dataset in English and Konkani
This work addresses the need for evaluating socio-cultural bias at subnational levels, specifically for the hyperlocal Goan identity groups, which is often overlooked in NLP fairness research.
The authors introduce AmchiBias, the first benchmark for measuring socio-cultural stereotypical bias for the Indian state of Goa, comprising 313 minimal pairs in English and Konkani. Evaluating five multilingual encoder models, they find near-chance scores in Konkani and higher bias for pan-Indian groups in English, highlighting a gap in low-resource multilingual NLP for hyperlocal identities.
Socio-cultural stereotypical bias is an important consideration in the development and deployment of NLP systems. It is however often considered only at the national level, despite rich subnational socio-cultural structures. We present AmchiBias, the first benchmark for measuring socio-cultural stereotypical bias for the Indian state of Goa with its unique historically multicultural setting. It covers various Goan identity groups and comprises 313 minimal pairs across eight sociodemographic dimensions in both English and Devanagari Konkani. We then evaluate stereotypical bias in five multilingual encoder models on this benchmark. We find near-chance scores in Konkani, reflecting language incompetence for general multilingual models and a lack of Goan cultural competence for Indian language models. Queried in English, models with a stronger Indian language coverage show higher bias for pan-Indian groups than hyperlocal Goan groups. This suggests the English signal reflects pan-Indian pretraining associations rather than genuine Goan cultural knowledge. Our findings highlight a critical gap in low-resource multilingual NLP evaluation for hyperlocal community identities.