Visual Semantic Description Generation with MLLMs for Image-Text Matching
This work addresses the modality gap in image-text matching for applications like retrieval and captioning, offering an incremental improvement by integrating MLLMs into existing models.
The paper tackles the problem of aligning visual and textual modalities in image-text matching by using multimodal large language models to generate visual semantic descriptions, achieving substantial performance improvements on Flickr30K and MSCOCO datasets with notable zero-shot generalization to cross-domain tasks.
Image-text matching (ITM) aims to address the fundamental challenge of aligning visual and textual modalities, which inherently differ in their representations, continuous, high-dimensional image features vs. discrete, structured text. We propose a novel framework that bridges the modality gap by leveraging multimodal large language models (MLLMs) as visual semantic parsers. By generating rich Visual Semantic Descriptions (VSD), MLLMs provide semantic anchor that facilitate cross-modal alignment. Our approach combines: (1) Instance-level alignment by fusing visual features with VSD to enhance the linguistic expressiveness of image representations, and (2) Prototype-level alignment through VSD clustering to ensure category-level consistency. These modules can be seamlessly integrated into existing ITM models. Extensive experiments on Flickr30K and MSCOCO demonstrate substantial performance improvements. The approach also exhibits remarkable zero-shot generalization to cross-domain tasks, including news and remote sensing ITM. The code and model checkpoints are available at https://github.com/Image-Text-Matching/VSD.