PoI: A Filter to Extract Pixel of Interest from Novel View Synthesis for Scene Coordinate Regression
This addresses a critical limitation for SCR methods in visual localization, particularly in sparse input scenarios, though it is incremental as it builds on existing NVS and SCR techniques.
The paper tackles the problem of rendering artifacts in novel view synthesis (NVS) images undermining training for scene coordinate regression (SCR) in camera pose estimation, proposing a dual-criteria filter to discard suboptimal pixels, which achieves state-of-the-art localization accuracy with computational efficiency.
Novel View synthesis (NVS) techniques, notably Neural Radiance Fields (NeRF) and 3D Gaussian Splatting (3DGS), can augment camera pose estimation by extending training data with rendered images. However, the images rendered by these methods are often plagued by blurring, undermining their reliability as training data for camera pose estimation. This limitation is particularly critical for Scene Coordinate Regression (SCR) methods, which aim at pixel-level 3D coordinate estimation, because rendering artifacts directly lead to estimation inaccuracies. To address this challenge, we propose a dual-criteria filtering mechanism that dynamically identifies and discards suboptimal pixels during training. The dual-criteria filter evaluates two concurrent metrics: (1) real-time SCR reprojection error, and (2) gradient threshold, across the coordinate regression domain. In addition, for visual localization problems in sparse input scenarios, it will be even more necessary to use data generated by NVS to assist the localization task. We design a coarse-to-fine PoI variant using sparse input NVS to solve this problem. Experiments across indoor and outdoor benchmarks confirm our method's efficacy. It achieves state-of-the-art localization accuracy while maintaining computational efficiency.