Multilingual Visual Sentiment Concept Matching
This work addresses the challenge of understanding cultural influences in visual sentiment for multimedia research, providing tools for retrieval and analysis, but it is incremental as it builds on existing methods for concept representation and clustering.
The study tackled the problem of cultural differences in visual emotion perception by developing computational tools to analyze a dataset of 16K multilingual affective visual concepts and 7.3M Flickr images, using crowdsourced sentiment judgments and word embeddings to explore commonalities and differences across languages.
The impact of culture in visual emotion perception has recently captured the attention of multimedia research. In this study, we pro- vide powerful computational linguistics tools to explore, retrieve and browse a dataset of 16K multilingual affective visual concepts and 7.3M Flickr images. First, we design an effective crowdsourc- ing experiment to collect human judgements of sentiment connected to the visual concepts. We then use word embeddings to repre- sent these concepts in a low dimensional vector space, allowing us to expand the meaning around concepts, and thus enabling insight about commonalities and differences among different languages. We compare a variety of concept representations through a novel evaluation task based on the notion of visual semantic relatedness. Based on these representations, we design clustering schemes to group multilingual visual concepts, and evaluate them with novel metrics based on the crowdsourced sentiment annotations as well as visual semantic relatedness. The proposed clustering framework enables us to analyze the full multilingual dataset in-depth and also show an application on a facial data subset, exploring cultural in- sights of portrait-related affective visual concepts.