Preserving Semantic Neighborhoods for Robust Cross-modal Retrieval
This addresses robust cross-modal retrieval for applications like social media and news, but is incremental as it builds on existing metric learning approaches.
The paper tackled the problem of cross-modal retrieval for real-world image-text pairs with complementary information, proposing within-modality losses to preserve semantic neighborhoods, and improved results on four datasets compared to five baselines.
The abundance of multimodal data (e.g. social media posts) has inspired interest in cross-modal retrieval methods. Popular approaches rely on a variety of metric learning losses, which prescribe what the proximity of image and text should be, in the learned space. However, most prior methods have focused on the case where image and text convey redundant information; in contrast, real-world image-text pairs convey complementary information with little overlap. Further, images in news articles and media portray topics in a visually diverse fashion; thus, we need to take special care to ensure a meaningful image representation. We propose novel within-modality losses which encourage semantic coherency in both the text and image subspaces, which does not necessarily align with visual coherency. Our method ensures that not only are paired images and texts close, but the expected image-image and text-text relationships are also observed. Our approach improves the results of cross-modal retrieval on four datasets compared to five baselines.