ADVISE: Symbolism and External Knowledge for Decoding Advertisements
This work addresses the challenge of interpreting advertisements for applications in marketing and AI, but it is incremental as it builds on existing object recognition and captioning methods.
The paper tackles the problem of understanding advertisements by leveraging symbolism and external knowledge, and shows that their method outperforms state-of-the-art on ad understanding and question-answering tasks.
In order to convey the most content in their limited space, advertisements embed references to outside knowledge via symbolism. For example, a motorcycle stands for adventure (a positive property the ad wants associated with the product being sold), and a gun stands for danger (a negative property to dissuade viewers from undesirable behaviors). We show how to use symbolic references to better understand the meaning of an ad. We further show how anchoring ad understanding in general-purpose object recognition and image captioning improves results. We formulate the ad understanding task as matching the ad image to human-generated statements that describe the action that the ad prompts, and the rationale it provides for taking this action. Our proposed method outperforms the state of the art on this task, and on an alternative formulation of question-answering on ads. We show additional applications of our learned representations for matching ads to slogans, and clustering ads according to their topic, without extra training.