On the Cone Effect and Modality Gap in Medical Vision-Language Embeddings
This work addresses the practical effects of the modality gap for researchers and practitioners in medical AI, offering a tunable approach to optimize multimodal representations, though it is incremental as it builds on known phenomena without introducing a new paradigm.
The study tackled the impact of the modality gap in vision-language models on supervised multimodal learning, particularly in medical domains, by introducing a lightweight post-hoc mechanism to control cross-modal separation, finding that reducing excessive gaps improves performance with medical datasets showing stronger sensitivity, though intermediate separation yields the best results.
Vision-Language Models (VLMs) exhibit a characteristic "cone effect" in which nonlinear encoders map embeddings into highly concentrated regions of the representation space, contributing to cross-modal separation known as the modality gap. While this phenomenon has been widely observed, its practical impact on supervised multimodal learning -particularly in medical domains- remains unclear. In this work, we introduce a lightweight post-hoc mechanism that keeps pretrained VLM encoders frozen while continuously controlling cross-modal separation through a single hyperparameter {λ}. This enables systematic analysis of how the modality gap affects downstream multimodal performance without expensive retraining. We evaluate generalist (CLIP, SigLIP) and medically specialized (BioMedCLIP, MedSigLIP) models across diverse medical and natural datasets in a supervised multimodal settings. Results consistently show that reducing excessive modality gap improves downstream performance, with medical datasets exhibiting stronger sensitivity to gap modulation; however, fully collapsing the gap is not always optimal, and intermediate, task-dependent separation yields the best results. These findings position the modality gap as a tunable property of multimodal representations rather than a quantity that should be universally minimized.