Mitigating Test-Time Bias for Fair Image Retrieval
This addresses fairness issues in image retrieval for users by mitigating bias in neutral queries, though it is incremental as it builds on existing bias-mitigation approaches.
The paper tackles test-time bias in image retrieval by introducing Post-hoc Bias Mitigation (PBM), a post-processing technique that reduces bias while maintaining retrieval performance, achieving the lowest bias compared to existing methods on datasets like Occupation 1 and 2, MS-COCO, and Flickr30k.
We address the challenge of generating fair and unbiased image retrieval results given neutral textual queries (with no explicit gender or race connotations), while maintaining the utility (performance) of the underlying vision-language (VL) model. Previous methods aim to disentangle learned representations of images and text queries from gender and racial characteristics. However, we show these are inadequate at alleviating bias for the desired equal representation result, as there usually exists test-time bias in the target retrieval set. So motivated, we introduce a straightforward technique, Post-hoc Bias Mitigation (PBM), that post-processes the outputs from the pre-trained vision-language model. We evaluate our algorithm on real-world image search datasets, Occupation 1 and 2, as well as two large-scale image-text datasets, MS-COCO and Flickr30k. Our approach achieves the lowest bias, compared with various existing bias-mitigation methods, in text-based image retrieval result while maintaining satisfactory retrieval performance. The source code is publicly available at \url{https://anonymous.4open.science/r/Fair_Text_based_Image_Retrieval-D8B2}.