Deep Permutation Equivariant Structure from Motion
This addresses the challenge of deep learning-based Structure from Motion for researchers and practitioners in computer vision, offering a novel approach that is incremental in combining existing ideas into a new architecture.
The paper tackles the problem of simultaneously recovering camera poses and 3D scene structure from multiple images without requiring initialization, using a neural network that respects permutation equivariance. The method achieves accuracy on par with classical state-of-the-art methods in both calibrated and uncalibrated settings, and a pre-trained network can reconstruct novel scenes with inexpensive fine-tuning without loss of accuracy.
Existing deep methods produce highly accurate 3D reconstructions in stereo and multiview stereo settings, i.e., when cameras are both internally and externally calibrated. Nevertheless, the challenge of simultaneous recovery of camera poses and 3D scene structure in multiview settings with deep networks is still outstanding. Inspired by projective factorization for Structure from Motion (SFM) and by deep matrix completion techniques, we propose a neural network architecture that, given a set of point tracks in multiple images of a static scene, recovers both the camera parameters and a (sparse) scene structure by minimizing an unsupervised reprojection loss. Our network architecture is designed to respect the structure of the problem: the sought output is equivariant to permutations of both cameras and scene points. Notably, our method does not require initialization of camera parameters or 3D point locations. We test our architecture in two setups: (1) single scene reconstruction and (2) learning from multiple scenes. Our experiments, conducted on a variety of datasets in both internally calibrated and uncalibrated settings, indicate that our method accurately recovers pose and structure, on par with classical state of the art methods. Additionally, we show that a pre-trained network can be used to reconstruct novel scenes using inexpensive fine-tuning with no loss of accuracy.