SEM: Sparse Embedding Modulation for Post-Hoc Debiasing of Vision-Language Models
This addresses bias issues in multimodal AI systems, which is crucial for fair deployment, but it is incremental as it builds on existing post-hoc debiasing methods by using sparse representations.
The paper tackled the problem of social and spurious biases in vision-language models like CLIP by proposing Sparse Embedding Modulation (SEM), a post-hoc debiasing framework that operates in a sparse latent space, achieving substantial fairness gains in retrieval and zero-shot classification across four benchmark datasets and two CLIP backbones.
Models that bridge vision and language, such as CLIP, are key components of multimodal AI, yet their large-scale, uncurated training data introduce severe social and spurious biases. Existing post-hoc debiasing methods often operate directly in the dense CLIP embedding space, where bias and task-relevant information are highly entangled. This entanglement limits their ability to remove bias without degrading semantic fidelity. In this work, we propose Sparse Embedding Modulation (SEM), a post-hoc, zero-shot debiasing framework that operates in a Sparse Autoencoder (SAE) latent space. By decomposing CLIP text embeddings into disentangled features, SEM identifies and modulates bias-relevant neurons while preserving query-relevant ones. This enables more precise, non-linear interventions. Across four benchmark datasets and two CLIP backbones, SEM achieves substantial fairness gains in retrieval and zero-shot classification. Our results demonstrate that sparse latent representations provide an effective foundation for post-hoc debiasing of vision-language models.