Vid2CAD: CAD Model Alignment using Multi-View Constraints from Videos
This work provides a significant improvement in 3D object pose estimation for CAD models from videos, which is beneficial for applications requiring accurate 3D scene reconstruction for robotics and augmented reality.
This paper tackles the problem of aligning CAD models to video sequences of complex scenes, automatically recovering the 9 DoF pose for each object. By integrating neural network predictions from individual frames with a global multi-view constraint optimization, the method achieves a substantial improvement in class average accuracy on the Scan2CAD dataset, increasing it from 11.6% to 30.7% compared to a state-of-the-art single-frame method.
We address the task of aligning CAD models to a video sequence of a complex scene containing multiple objects. Our method can process arbitrary videos and fully automatically recover the 9 DoF pose for each object appearing in it, thus aligning them in a common 3D coordinate frame. The core idea of our method is to integrate neural network predictions from individual frames with a temporally global, multi-view constraint optimization formulation. This integration process resolves the scale and depth ambiguities in the per-frame predictions, and generally improves the estimate of all pose parameters. By leveraging multi-view constraints, our method also resolves occlusions and handles objects that are out of view in individual frames, thus reconstructing all objects into a single globally consistent CAD representation of the scene. In comparison to the state-of-the-art single-frame method Mask2CAD that we build on, we achieve substantial improvements on the Scan2CAD dataset (from 11.6% to 30.7% class average accuracy).