Probing Gender Bias in Multilingual LLMs: A Case Study of Stereotypes in Persian
This addresses the problem of representational harm from gender bias in LLMs for low-resource language users, providing a framework for assessment, but it is incremental as it extends existing bias studies to a new language.
The study tackled gender bias in multilingual LLMs by evaluating four models (GPT-4o mini, DeepSeek R1, Gemini 2.0 Flash, Qwen QwQ 32B) in Persian, a low-resource language, and found that all models exhibited gender stereotypes, with greater disparities in Persian than in English, especially in sports.
Multilingual Large Language Models (LLMs) are increasingly used worldwide, making it essential to ensure they are free from gender bias to prevent representational harm. While prior studies have examined such biases in high-resource languages, low-resource languages remain understudied. In this paper, we propose a template-based probing methodology, validated against real-world data, to uncover gender stereotypes in LLMs. As part of this framework, we introduce the Domain-Specific Gender Skew Index (DS-GSI), a metric that quantifies deviations from gender parity. We evaluate four prominent models, GPT-4o mini, DeepSeek R1, Gemini 2.0 Flash, and Qwen QwQ 32B, across four semantic domains, focusing on Persian, a low-resource language with distinct linguistic features. Our results show that all models exhibit gender stereotypes, with greater disparities in Persian than in English across all domains. Among these, sports reflect the most rigid gender biases. This study underscores the need for inclusive NLP practices and provides a framework for assessing bias in other low-resource languages.