RelPose: Predicting Probabilistic Relative Rotation for Single Objects in the Wild
This work addresses a core challenge in geometric pipelines and neural reconstruction methods, offering a stepping stone for in-the-wild object reconstruction from multi-view datasets, though it appears incremental as it builds on existing viewpoint estimation tasks.
The paper tackles the problem of estimating camera viewpoints from sparse images of arbitrary objects, proposing a top-down, energy-based method that explicitly models multimodal distributions over relative rotations. The approach outperforms state-of-the-art SfM and SLAM methods on both seen and unseen categories, with significant gains over direct regression, demonstrating the importance of modeling multimodality for coherent joint reconstruction.
We describe a data-driven method for inferring the camera viewpoints given multiple images of an arbitrary object. This task is a core component of classic geometric pipelines such as SfM and SLAM, and also serves as a vital pre-processing requirement for contemporary neural approaches (e.g. NeRF) to object reconstruction and view synthesis. In contrast to existing correspondence-driven methods that do not perform well given sparse views, we propose a top-down prediction based approach for estimating camera viewpoints. Our key technical insight is the use of an energy-based formulation for representing distributions over relative camera rotations, thus allowing us to explicitly represent multiple camera modes arising from object symmetries or views. Leveraging these relative predictions, we jointly estimate a consistent set of camera rotations from multiple images. We show that our approach outperforms state-of-the-art SfM and SLAM methods given sparse images on both seen and unseen categories. Further, our probabilistic approach significantly outperforms directly regressing relative poses, suggesting that modeling multimodality is important for coherent joint reconstruction. We demonstrate that our system can be a stepping stone toward in-the-wild reconstruction from multi-view datasets. The project page with code and videos can be found at https://jasonyzhang.com/relpose.