Not just a matter of semantics: the relationship between visual similarity and semantic similarity
This work addresses a foundational assumption in computer vision and machine learning, with implications for knowledge transfer and semantic image retrieval, though it is incremental in analyzing existing methods.
The paper investigates the relationship between visual and semantic similarity, challenging the assumption that semantic similarity correlates with visual similarity, which is crucial for methods like zero-shot learning. Results show that WordNet semantic similarity provides more information than just class differences, but classification is not ideal for semantic methods, and incorrect semantic information can be harmful.
Knowledge transfer, zero-shot learning and semantic image retrieval are methods that aim at improving accuracy by utilizing semantic information, e.g. from WordNet. It is assumed that this information can augment or replace missing visual data in the form of labeled training images because semantic similarity correlates with visual similarity. This assumption may seem trivial, but is crucial for the application of such semantic methods. Any violation can cause mispredictions. Thus, it is important to examine the visual-semantic relationship for a certain target problem. In this paper, we use five different semantic and visual similarity measures each to thoroughly analyze the relationship without relying too much on any single definition. We postulate and verify three highly consequential hypotheses on the relationship. Our results show that it indeed exists and that WordNet semantic similarity carries more information about visual similarity than just the knowledge of "different classes look different". They suggest that classification is not the ideal application for semantic methods and that wrong semantic information is much worse than none.