"Is this an example image?" -- Predicting the Relative Abstractness Level of Image and Text
This work addresses a specific challenge in multimodal search and retrieval for researchers, but it is incremental as it builds on existing metrics like cross-modal mutual information.
The paper tackles the problem of predicting the relative abstractness level between image and text pairs, introducing a new metric (ABS) and a deep learning approach that reduces the need for labeled data, with experimental results showing feasibility on a challenging test set.
Successful multimodal search and retrieval requires the automatic understanding of semantic cross-modal relations, which, however, is still an open research problem. Previous work has suggested the metrics cross-modal mutual information and semantic correlation to model and predict cross-modal semantic relations of image and text. In this paper, we present an approach to predict the (cross-modal) relative abstractness level of a given image-text pair, that is whether the image is an abstraction of the text or vice versa. For this purpose, we introduce a new metric that captures this specific relationship between image and text at the Abstractness Level (ABS). We present a deep learning approach to predict this metric, which relies on an autoencoder architecture that allows us to significantly reduce the required amount of labeled training data. A comprehensive set of publicly available scientific documents has been gathered. Experimental results on a challenging test set demonstrate the feasibility of the approach.