In-Hand 3D Object Scanning from an RGB Sequence
This addresses the challenge of in-hand 3D scanning for robotics or AR/VR applications, but it is incremental as it builds on existing NeRF-based approaches.
The paper tackles the problem of 3D scanning an unknown object held in hand using a monocular camera without known camera-object poses, by optimizing shape and pose simultaneously with an incremental neural implicit method, achieving performance close to methods that assume known poses.
We propose a method for in-hand 3D scanning of an unknown object with a monocular camera. Our method relies on a neural implicit surface representation that captures both the geometry and the appearance of the object, however, by contrast with most NeRF-based methods, we do not assume that the camera-object relative poses are known. Instead, we simultaneously optimize both the object shape and the pose trajectory. As direct optimization over all shape and pose parameters is prone to fail without coarse-level initialization, we propose an incremental approach that starts by splitting the sequence into carefully selected overlapping segments within which the optimization is likely to succeed. We reconstruct the object shape and track its poses independently within each segment, then merge all the segments before performing a global optimization. We show that our method is able to reconstruct the shape and color of both textured and challenging texture-less objects, outperforms classical methods that rely only on appearance features, and that its performance is close to recent methods that assume known camera poses.