Visual Re-Ranking with Non-Visual Side Information
This work addresses the challenge of enhancing visual place recognition accuracy for applications like robotics and augmented reality by leveraging readily available side information, representing an incremental advance over existing re-ranking methods.
The paper tackles the problem of limited improvement in visual place recognition from re-ranking methods that rely solely on visual descriptors by proposing GCSA, a graph neural network-based re-ranking method that incorporates non-visual side information like sensor data or camera poses. The result shows significant improvements in image retrieval metrics and visual localization tasks on large-scale datasets.
The standard approach for visual place recognition is to use global image descriptors to retrieve the most similar database images for a given query image. The results can then be further improved with re-ranking methods that re-order the top scoring images. However, existing methods focus on re-ranking based on the same image descriptors that were used for the initial retrieval, which we argue provides limited additional signal. In this work we propose Generalized Contextual Similarity Aggregation (GCSA), which is a graph neural network-based re-ranking method that, in addition to the visual descriptors, can leverage other types of available side information. This can for example be other sensor data (such as signal strength of nearby WiFi or BlueTooth endpoints) or geometric properties such as camera poses for database images. In many applications this information is already present or can be acquired with low effort. Our architecture leverages the concept of affinity vectors to allow for a shared encoding of the heterogeneous multi-modal input. Two large-scale datasets, covering both outdoor and indoor localization scenarios, are utilized for training and evaluation. In experiments we show significant improvement not only on image retrieval metrics, but also for the downstream visual localization task.