Pixel-Perfect Structure-from-Motion with Featuremetric Refinement
This addresses the issue of error propagation in structure-from-motion for applications like crowd-sourced localization, though it is incremental as it builds on existing SfM methods.
The paper tackles the problem of poorly-localized features in sparse 3D reconstruction by refining keypoint locations and camera poses using featuremetric alignment, resulting in significantly improved accuracy for camera poses and scene geometry across various conditions.
Finding local features that are repeatable across multiple views is a cornerstone of sparse 3D reconstruction. The classical image matching paradigm detects keypoints per-image once and for all, which can yield poorly-localized features and propagate large errors to the final geometry. In this paper, we refine two key steps of structure-from-motion by a direct alignment of low-level image information from multiple views: we first adjust the initial keypoint locations prior to any geometric estimation, and subsequently refine points and camera poses as a post-processing. This refinement is robust to large detection noise and appearance changes, as it optimizes a featuremetric error based on dense features predicted by a neural network. This significantly improves the accuracy of camera poses and scene geometry for a wide range of keypoint detectors, challenging viewing conditions, and off-the-shelf deep features. Our system easily scales to large image collections, enabling pixel-perfect crowd-sourced localization at scale. Our code is publicly available at https://github.com/cvg/pixel-perfect-sfm as an add-on to the popular SfM software COLMAP.