Nearest Neighbor Normalization Improves Multimodal Retrieval
This work addresses the issue of retrieval accuracy for users of multimodal models, but it is incremental as it builds on existing contrastive methods.
The paper tackles the problem of imperfect performance in multimodal retrieval by introducing Nearest Neighbor Normalization (NNN), a method that corrects errors in trained contrastive models without additional training, resulting in improved retrieval metrics across multiple models and datasets.
Multimodal models leverage large-scale pre-training to achieve strong but still imperfect performance on tasks such as image captioning, visual question answering, and cross-modal retrieval. In this paper, we present a simple and efficient method for correcting errors in trained contrastive image-text retrieval models with no additional training, called Nearest Neighbor Normalization (NNN). We show an improvement on retrieval metrics in both text retrieval and image retrieval for all of the contrastive models that we tested (CLIP, BLIP, ALBEF, SigLIP, BEiT) and for both of the datasets that we used (MS-COCO and Flickr30k). NNN requires a reference database, but does not require any training on this database, and can even increase the retrieval accuracy of a model after finetuning.