PIZZA: A Powerful Image-only Zero-Shot Zero-CAD Approach to 6 DoF Tracking
This enables instant tracking of unknown objects in robotics and AR without prior data, though it is incremental as it matches rather than surpasses existing methods.
The paper tackles the problem of tracking 6D motion of unknown objects in RGB video without prior training images or 3D models, achieving results on par with methods that require more information.
Estimating the relative pose of a new object without prior knowledge is a hard problem, while it is an ability very much needed in robotics and Augmented Reality. We present a method for tracking the 6D motion of objects in RGB video sequences when neither the training images nor the 3D geometry of the objects are available. In contrast to previous works, our method can therefore consider unknown objects in open world instantly, without requiring any prior information or a specific training phase. We consider two architectures, one based on two frames, and the other relying on a Transformer Encoder, which can exploit an arbitrary number of past frames. We train our architectures using only synthetic renderings with domain randomization. Our results on challenging datasets are on par with previous works that require much more information (training images of the target objects, 3D models, and/or depth data). Our source code is available at https://github.com/nv-nguyen/pizza