Uncovering Visual-Semantic Psycholinguistic Properties from the Distributional Structure of Text Embedding Space
This work addresses the problem of linking visual and semantic spaces for researchers in computational linguistics and psychology, but it is incremental as it builds on existing unsupervised approaches.
The paper tackled estimating psycholinguistic properties like imageability and concreteness from text embeddings, showing that their proposed Neighborhood Stability Measure (NSM) correlates more strongly with ground-truth ratings than existing unsupervised methods.
Imageability (potential of text to evoke a mental image) and concreteness (perceptibility of text) are two psycholinguistic properties that link visual and semantic spaces. It is little surprise that computational methods that estimate them do so using parallel visual and semantic spaces, such as collections of image-caption pairs or multi-modal models. In this paper, we work on the supposition that text itself in an image-caption dataset offers sufficient signals to accurately estimate these properties. We hypothesize, in particular, that the peakedness of the neighborhood of a word in the semantic embedding space reflects its degree of imageability and concreteness. We then propose an unsupervised, distribution-free measure, which we call Neighborhood Stability Measure (NSM), that quantifies the sharpness of peaks. Extensive experiments show that NSM correlates more strongly with ground-truth ratings than existing unsupervised methods, and is a strong predictor of these properties for classification. Our code and data are available on GitHub (https://github.com/Artificial-Memory-Lab/imageability).