SeeGULL Multilingual: a Dataset of Geo-Culturally Situated Stereotypes
This addresses the problem of limited safety and fairness evaluations for non-English languages in AI, though it is incremental as it extends an existing approach to a broader scale.
The authors tackled the lack of multilingual resources for evaluating stereotypes in generative models by building SeeGULL Multilingual, a dataset with over 25K stereotypes across 20 languages and 23 regions, which helps identify gaps in model evaluations.
While generative multilingual models are rapidly being deployed, their safety and fairness evaluations are largely limited to resources collected in English. This is especially problematic for evaluations targeting inherently socio-cultural phenomena such as stereotyping, where it is important to build multi-lingual resources that reflect the stereotypes prevalent in respective language communities. However, gathering these resources, at scale, in varied languages and regions pose a significant challenge as it requires broad socio-cultural knowledge and can also be prohibitively expensive. To overcome this critical gap, we employ a recently introduced approach that couples LLM generations for scale with culturally situated validations for reliability, and build SeeGULL Multilingual, a global-scale multilingual dataset of social stereotypes, containing over 25K stereotypes, spanning 20 languages, with human annotations across 23 regions, and demonstrate its utility in identifying gaps in model evaluations. Content warning: Stereotypes shared in this paper can be offensive.