Content-based Image Retrieval and the Semantic Gap in the Deep Learning Era
This work highlights a critical gap in image retrieval research, pointing out that current methods fail in semantic tasks, which is important for applications needing deeper image understanding.
The paper investigates whether advances in instance retrieval transfer to semantic image retrieval, finding that instance retrieval methods perform worse than simpler generic methods in tasks requiring image understanding, and identifies a lack of standardized benchmarks as a key barrier.
Content-based image retrieval has seen astonishing progress over the past decade, especially for the task of retrieving images of the same object that is depicted in the query image. This scenario is called instance or object retrieval and requires matching fine-grained visual patterns between images. Semantics, however, do not play a crucial role. This brings rise to the question: Do the recent advances in instance retrieval transfer to more generic image retrieval scenarios? To answer this question, we first provide a brief overview of the most relevant milestones of instance retrieval. We then apply them to a semantic image retrieval task and find that they perform inferior to much less sophisticated and more generic methods in a setting that requires image understanding. Following this, we review existing approaches to closing this so-called semantic gap by integrating prior world knowledge. We conclude that the key problem for the further advancement of semantic image retrieval lies in the lack of a standardized task definition and an appropriate benchmark dataset.