SegDebias: Test-Time Bias Mitigation for ViT-Based CLIP via Segmentation
This addresses bias mitigation in vision-language models for real-world applications without needing training data or bias annotations, though it is incremental as it builds on existing test-time methods.
The paper tackled the problem of spurious correlations in CLIP models by proposing a test-time debiasing method that uses segmentation to isolate target attributes, resulting in improved group robustness metrics and Attention IoU on datasets like Waterbirds and CelebA.
Vision language models such as CLIP have shown remarkable performance in zero shot classification, but remain susceptible to spurious correlations, where irrelevant visual features influence predictions. Existing debiasing methods often require access to training data and explicit group labels to perform fine-tuning or adjust embeddings, which limits their practicality in real-world settings. Test-time methods attempt to avoid this constraint, but many still depend on prior knowledge of dataset specific biases, limiting their generalizability in open set settings. In this work, we propose a test-time debiasing method for ViT based CLIP models that requires no additional training or assumptions of bias annotations. Our approach uses a pretrained segmentation model to isolate the target visual attribute, then adjusts the non target regions so that their embeddings are uniformly similar to all class specific text prompts. This procedure removes unintended bias signals from confounding visual regions while preserving the target attribute. Experiments on Waterbirds and CelebA show that our method outperforms existing test-time debiasing approaches in both group robustness metrics and Attention IoU. These results demonstrate the effectiveness of segmentation guided interventions for scalable and annotation free bias mitigation in vision language models.