Learning Deep Structure-Preserving Image-Text Embeddings
This work addresses cross-modal retrieval for applications like image search and captioning, representing an incremental improvement over existing methods.
The paper tackles the problem of learning joint embeddings for images and text to improve cross-modal retrieval, achieving new state-of-the-art results on Flickr30K and MSCOCO datasets with significant accuracy gains.
This paper proposes a method for learning joint embeddings of images and text using a two-branch neural network with multiple layers of linear projections followed by nonlinearities. The network is trained using a large margin objective that combines cross-view ranking constraints with within-view neighborhood structure preservation constraints inspired by metric learning literature. Extensive experiments show that our approach gains significant improvements in accuracy for image-to-text and text-to-image retrieval. Our method achieves new state-of-the-art results on the Flickr30K and MSCOCO image-sentence datasets and shows promise on the new task of phrase localization on the Flickr30K Entities dataset.