On Train-Test Class Overlap and Detection for Image Retrieval
This addresses the issue of dataset bias for researchers and practitioners in image retrieval, but it is incremental as it builds on existing benchmarks and methods.
The study tackled the problem of class overlap between training and evaluation sets in image retrieval, finding that removing overlap from Google Landmarks v2 clean caused a dramatic and inconsistent performance drop across methods. They introduced CiDeR, an end-to-end pipeline that outperformed previous state-of-the-art on both existing and new datasets.
How important is it for training and evaluation sets to not have class overlap in image retrieval? We revisit Google Landmarks v2 clean, the most popular training set, by identifying and removing class overlap with Revisited Oxford and Paris [34], the most popular evaluation set. By comparing the original and the new RGLDv2-clean on a benchmark of reproduced state-of-the-art methods, our findings are striking. Not only is there a dramatic drop in performance, but it is inconsistent across methods, changing the ranking.What does it take to focus on objects or interest and ignore background clutter when indexing? Do we need to train an object detector and the representation separately? Do we need location supervision? We introduce Single-stage Detect-to-Retrieve (CiDeR), an end-to-end, single-stage pipeline to detect objects of interest and extract a global image representation. We outperform previous state-of-the-art on both existing training sets and the new RGLDv2-clean. Our dataset is available at https://github.com/dealicious-inc/RGLDv2-clean.