Beyond Symmetric Alignment: Spectral Diagnostics of Modality Imbalance in Vision-Language Models in the Medical Domain
For researchers and practitioners deploying VLMs in medical imaging, SAS provides a practical diagnostic tool to identify which modality drives cross-modal degradation, addressing a critical bottleneck in clinical VLM applications.
The paper introduces the Spectral Alignment Score (SAS), an asymmetric metric that diagnoses modality imbalance in vision-language models (VLMs) by revealing that medical images retain richer structural information than their paired clinical reports. SAS achieves the strongest zero-label correlation with retrieval performance in the medical domain, outperforming existing symmetric metrics.
Vision-Language Models (VLMs) struggle when applied to medical image-text data, yet the tools available to diagnose this failure remain limited. Existing representation alignment metrics are symmetric, collapsing both modalities into a single score and hiding which modality drives cross-modal degradation. We introduce the Spectral Alignment Score (SAS), an asymmetric metric that projects both modalities onto the principal eigenbasis of an anchor modality and computes eigenvalue-weighted per-eigenmode correlations, resulting in directional scores whose difference quantifies modality information imbalance. We embed SAS within a benchmarking framework evaluating 15 VLMs across natural and medical image-text datasets alongside 6 alignment metrics and bidirectional retrieval. Our experiments show that medical images retain richer structural information than their paired clinical reports, a directional asymmetry invisible to all competing metrics, and that SAS achieves the strongest zero-label correlation with retrieval performance in the medical domain, positioning it as a practical diagnostic tool for clinical deployment. Code is available at this URL: https://github.com/iamalegambetti/medical-vlms-assessment.