D2S: Representing sparse descriptors and 3D coordinates for camera relocalization
This work addresses efficiency and scalability issues in camera relocalization for robotics and AR/VR applications, offering a lightweight and cost-effective solution, though it is incremental as it builds on existing learning-based approaches.
The paper tackles the high computational and storage costs of visual localization by proposing D2S, a direct learning-based method that uses a simple network to represent sparse descriptors and 3D coordinates from a single RGB image, outperforming previous regression-based methods in indoor and outdoor environments with generalization to day-night transitions and domain shifts.
State-of-the-art visual localization methods mostly rely on complex procedures to match local descriptors and 3D point clouds. However, these procedures can incur significant costs in terms of inference, storage, and updates over time. In this study, we propose a direct learning-based approach that utilizes a simple network named D2S to represent complex local descriptors and their scene coordinates. Our method is characterized by its simplicity and cost-effectiveness. It solely leverages a single RGB image for localization during the testing phase and only requires a lightweight model to encode a complex sparse scene. The proposed D2S employs a combination of a simple loss function and graph attention to selectively focus on robust descriptors while disregarding areas such as clouds, trees, and several dynamic objects. This selective attention enables D2S to effectively perform a binary-semantic classification for sparse descriptors. Additionally, we propose a simple outdoor dataset to evaluate the capabilities of visual localization methods in scene-specific generalization and self-updating from unlabeled observations. Our approach outperforms the previous regression-based methods in both indoor and outdoor environments. It demonstrates the ability to generalize beyond training data, including scenarios involving transitions from day to night and adapting to domain shifts. The source code, trained models, dataset, and demo videos are available at the following link: https://thpjp.github.io/d2s.