A Model for Multi-View Residual Covariances based on Perspective Deformation
This work addresses a specific bottleneck in computer vision for robotics and 3D reconstruction, offering incremental improvements in accuracy and state estimation.
The paper tackles the problem of modeling residual covariances in multi-view SfM, odometry, and SLAM by incorporating perspective deformation effects, resulting in improved accuracy for both feature-based and direct methods with negligible overhead.
In this work, we derive a model for the covariance of the visual residuals in multi-view SfM, odometry and SLAM setups. The core of our approach is the formulation of the residual covariances as a combination of geometric and photometric noise sources. And our key novel contribution is the derivation of a term modelling how local 2D patches suffer from perspective deformation when imaging 3D surfaces around a point. Together, these add up to an efficient and general formulation which not only improves the accuracy of both feature-based and direct methods, but can also be used to estimate more accurate measures of the state entropy and hence better founded point visibility thresholds. We validate our model with synthetic and real data and integrate it into photometric and feature-based Bundle Adjustment, improving their accuracy with a negligible overhead.