Equal But Not The Same: Understanding the Implicit Relationship Between Persuasive Images and Text
This work addresses the challenge of analyzing persuasive media for advertisers and researchers, but it is incremental as it builds on existing methods with a new dataset.
The paper tackled the problem of understanding complex, non-literal relationships between images and text in advertisements, showing that their method outperforms standard image-text alignment approaches in predicting parallel or non-parallel relationships.
Images and text in advertisements interact in complex, non-literal ways. The two channels are usually complementary, with each channel telling a different part of the story. Current approaches, such as image captioning methods, only examine literal, redundant relationships, where image and text show exactly the same content. To understand more complex relationships, we first collect a dataset of advertisement interpretations for whether the image and slogan in the same visual advertisement form a parallel (conveying the same message without literally saying the same thing) or non-parallel relationship, with the help of workers recruited on Amazon Mechanical Turk. We develop a variety of features that capture the creativity of images and the specificity or ambiguity of text, as well as methods that analyze the semantics within and across channels. We show that our method outperforms standard image-text alignment approaches on predicting the parallel/non-parallel relationship between image and text.