MAVias: Mitigate any Visual Bias
This addresses the limitation of existing bias mitigation methods that only handle predefined biases, which is crucial for building trustworthy AI in visual recognition applications.
The paper tackles the problem of mitigating multiple unknown biases in computer vision models by introducing MAVias, an open-set approach that uses foundation models to discover spurious visual associations and mitigate them during training, achieving state-of-the-art performance on diverse datasets like CelebA and ImageNet.
Mitigating biases in computer vision models is an essential step towards the trustworthiness of artificial intelligence models. Existing bias mitigation methods focus on a small set of predefined biases, limiting their applicability in visual datasets where multiple, possibly unknown biases exist. To address this limitation, we introduce MAVias, an open-set bias mitigation approach leveraging foundation models to discover spurious associations between visual attributes and target classes. MAVias first captures a wide variety of visual features in natural language via a foundation image tagging model, and then leverages a large language model to select those visual features defining the target class, resulting in a set of language-coded potential visual biases. We then translate this set of potential biases into vision-language embeddings and introduce an in-processing bias mitigation approach to prevent the model from encoding information related to them. Our experiments on diverse datasets, including CelebA, Waterbirds, ImageNet, and UrbanCars, show that MAVias effectively detects and mitigates a wide range of biases in visual recognition tasks outperforming current state-of-the-art.