A Prompt Array Keeps the Bias Away: Debiasing Vision-Language Models with Adversarial Learning
This addresses societal bias issues in multimodal AI systems, offering a debiasing solution for researchers and practitioners, though it is incremental as it builds on existing adversarial and contrastive techniques.
The paper tackled bias in vision-language models by developing a method that prepends learned embeddings to text queries, trained with adversarial debiasing and contrastive loss, which reduced various bias measures with minimal degradation to image-text representations.
Vision-language models can encode societal biases and stereotypes, but there are challenges to measuring and mitigating these multimodal harms due to lacking measurement robustness and feature degradation. To address these challenges, we investigate bias measures and apply ranking metrics for image-text representations. We then investigate debiasing methods and show that prepending learned embeddings to text queries that are jointly trained with adversarial debiasing and a contrastive loss reduces various bias measures with minimal degradation to the image-text representation.