Aligning Multilingual Word Embeddings for Cross-Modal Retrieval Task
This work addresses cross-modal retrieval for multilingual applications, but it is incremental as it combines existing objective functions.
The paper tackles the problem of matching images and captions across multiple languages by learning multimodal multilingual embeddings, achieving state-of-the-art performance in cross-modal retrieval tasks.
In this paper, we propose a new approach to learn multimodal multilingual embeddings for matching images and their relevant captions in two languages. We combine two existing objective functions to make images and captions close in a joint embedding space while adapting the alignment of word embeddings between existing languages in our model. We show that our approach enables better generalization, achieving state-of-the-art performance in text-to-image and image-to-text retrieval task, and caption-caption similarity task. Two multimodal multilingual datasets are used for evaluation: Multi30k with German and English captions and Microsoft-COCO with English and Japanese captions.