An Analysis of Social Biases Present in BERT Variants Across Multiple Languages
This work addresses the problem of social bias in language models for NLP practitioners and researchers, highlighting cultural and linguistic variations, but it is incremental as it extends existing bias analysis to more languages and dimensions.
The paper analyzed social biases in monolingual BERT models across English, Greek, and Persian, focusing on religious and ethnic biases beyond gender, and found that bias probing methods are highly language-dependent, with higher biases in non-English models potentially linked to user-generated training content.
Although large pre-trained language models have achieved great success in many NLP tasks, it has been shown that they reflect human biases from their pre-training corpora. This bias may lead to undesirable outcomes when these models are applied in real-world settings. In this paper, we investigate the bias present in monolingual BERT models across a diverse set of languages (English, Greek, and Persian). While recent research has mostly focused on gender-related biases, we analyze religious and ethnic biases as well and propose a template-based method to measure any kind of bias, based on sentence pseudo-likelihood, that can handle morphologically complex languages with gender-based adjective declensions. We analyze each monolingual model via this method and visualize cultural similarities and differences across different dimensions of bias. Ultimately, we conclude that current methods of probing for bias are highly language-dependent, necessitating cultural insights regarding the unique ways bias is expressed in each language and culture (e.g. through coded language, synecdoche, and other similar linguistic concepts). We also hypothesize that higher measured social biases in the non-English BERT models correlate with user-generated content in their training.