Grep-BiasIR: A Dataset for Investigating Gender Representation-Bias in Information Retrieval Results
This addresses the issue of societal bias reinforcement in IR systems for researchers and developers, but it is incremental as it provides a new dataset rather than a novel method.
The authors tackled the problem of gender bias in information retrieval systems by introducing Grep-BiasIR, a dataset with 118 bias-sensitive neutral search queries and accompanying documents in female, male, and neutral variations to facilitate bias studies.
The provided contents by information retrieval (IR) systems can reflect the existing societal biases and stereotypes. Such biases in retrieval results can lead to further establishing and strengthening stereotypes in society and also in the systems. To facilitate the studies of gender bias in the retrieval results of IR systems, we introduce Gender Representation-Bias for Information Retrieval (Grep-BiasIR), a novel thoroughly-audited dataset consisting of 118 bias-sensitive neutral search queries. The set of queries covers a wide range of gender-related topics, for which a biased representation of genders in the search result can be considered as socially problematic. Each query is accompanied with one relevant and one non-relevant document, where the document is also provided in three variations of female, male, and neutral. The dataset is available at https://github.com/KlaraKrieg/GrepBiasIR.