DualPM: Dual Posed-Canonical Point Maps for 3D Shape and Pose Reconstruction
This work addresses the problem of 3D analysis for deformable objects like quadrupeds, offering a novel representation that improves reconstruction accuracy, though it is incremental in extending point map concepts to a new domain.
The paper tackles 3D shape and pose reconstruction of deformable objects by introducing Dual Point Maps (DualPM), a representation that maps pixels to 3D object locations and a canonical rest pose, enabling amodal reconstruction. It demonstrates that this approach, trained on synthetic data with one or two models per category, generalizes to real images and achieves significant improvements over previous methods.
The choice of data representation is a key factor in the success of deep learning in geometric tasks. For instance, DUSt3R recently introduced the concept of viewpoint-invariant point maps, generalizing depth prediction and showing that all key problems in the 3D reconstruction of static scenes can be reduced to predicting such point maps. In this paper, we develop an analogous concept for a very different problem: the reconstruction of the 3D shape and pose of deformable objects. To this end, we introduce Dual Point Maps (DualPM), where a pair of point maps is extracted from the same image-one associating pixels to their 3D locations on the object and the other to a canonical version of the object in its rest pose. We also extend point maps to amodal reconstruction to recover the complete shape of the object, even through self-occlusions. We show that 3D reconstruction and 3D pose estimation can be reduced to the prediction of DualPMs. Empirically, we demonstrate that this representation is a suitable target for deep networks to predict. Specifically, we focus on modeling quadrupeds, showing that DualPMs can be trained purely on synthetic 3D data, consisting of one or two models per category, while generalizing effectively to real images. With this approach, we achieve significant improvements over previous methods for the 3D analysis and reconstruction of such objects.