SC-wLS: Towards Interpretable Feed-forward Camera Re-localization
This work addresses camera re-localization for robotics and augmented reality, offering an incremental improvement over existing feed-forward methods.
The paper tackles the problem of low accuracy in feed-forward camera re-localization by proposing SC-wLS, a method that uses weighted least squares on scene coordinates to improve performance, achieving significant gains on 7Scenes and Cambridge datasets.
Visual re-localization aims to recover camera poses in a known environment, which is vital for applications like robotics or augmented reality. Feed-forward absolute camera pose regression methods directly output poses by a network, but suffer from low accuracy. Meanwhile, scene coordinate based methods are accurate, but need iterative RANSAC post-processing, which brings challenges to efficient end-to-end training and inference. In order to have the best of both worlds, we propose a feed-forward method termed SC-wLS that exploits all scene coordinate estimates for weighted least squares pose regression. This differentiable formulation exploits a weight network imposed on 2D-3D correspondences, and requires pose supervision only. Qualitative results demonstrate the interpretability of learned weights. Evaluations on 7Scenes and Cambridge datasets show significantly promoted performance when compared with former feed-forward counterparts. Moreover, our SC-wLS method enables a new capability: self-supervised test-time adaptation on the weight network. Codes and models are publicly available.