Indexing Multimodal Language Models for Large-scale Image Retrieval
For practitioners in large-scale image retrieval, this work demonstrates that MLLMs can serve as effective zero-shot re-rankers without fine-tuning, offering a new alternative to specialized models.
The authors propose using Multimodal Large Language Models (MLLMs) as training-free similarity estimators for instance-level image retrieval, achieving superior robustness to clutter and occlusion compared to task-specific re-rankers, though performance degrades under severe appearance changes.
Multimodal Large Language Models (MLLMs) have demonstrated strong cross-modal reasoning capabilities, yet their potential for vision-only tasks remains underexplored. We investigate MLLMs as training-free similarity estimators for instance-level image-to-image retrieval. Our approach prompts the model with paired images and converts next-token probabilities into similarity scores, enabling zero-shot re-ranking within large-scale retrieval pipelines. This design avoids specialized architectures and fine-tuning, leveraging the rich visual discrimination learned during multimodal pre-training. We address scalability by combining MLLMs with memory-efficient indexing and top-$k$ candidate re-ranking. Experiments across diverse benchmarks show that MLLMs outperform task-specific re-rankers outside their native domains and exhibit superior robustness to clutter, occlusion, and small objects. Despite strong results, we identify failure modes under severe appearance changes, highlighting opportunities for future research. Our findings position MLLMs as a promising alternative for open-world large-scale image retrieval.