Is the Modality Gap a Bug or a Feature? A Robustness Perspective
For practitioners using vision-language models, this work provides a simple method to enhance model robustness, though the findings are incremental as they build on existing observations of the modality gap.
The paper explains the origin of the modality gap in multi-modal models like CLIP and shows that reducing this gap via post-processing improves robustness to embedding perturbations without sacrificing clean accuracy.
Many modern multi-modal models (e.g. CLIP) seek an embedding space in which the two modalities are aligned. Somewhat surprisingly, almost all existing models show a strong modality gap: the distribution of images is well-separated from the distribution of texts in the shared embedding space. Despite a series of recent papers on this topic, it is still not clear why this gap exists nor whether closing the gap in post-processing will lead to better performance on downstream tasks. In this paper we show that under certain conditions, minimizing the contrastive loss yields a representation in which the two modalities are separated by a global gap vector that is orthogonal to their embeddings. We also show that under these conditions the modality gap is monotonically related to robustness: decreasing the gap does not change the clean accuracy of the models but makes it less likely that a model will change its output when the embeddings are perturbed. Our experiments show that for many real-world VLMs we can significantly increase robustness by a simple post-processing step that moves one modality towards the mean of the other modality, without any loss of clean accuracy.