3D Implicit Transporter for Temporally Consistent Keypoint Discovery
This addresses the need for temporally consistent keypoints in 3D visual and robotic tasks, representing an incremental advancement by adapting a 2D method to 3D.
The paper tackles the problem of detecting keypoints in 3D point clouds by extending the Transporter method to incorporate temporal consistency, achieving spatio-temporally consistent keypoints on articulated objects and nonrigid animals, and demonstrates superior performance in 3D object manipulation.
Keypoint-based representation has proven advantageous in various visual and robotic tasks. However, the existing 2D and 3D methods for detecting keypoints mainly rely on geometric consistency to achieve spatial alignment, neglecting temporal consistency. To address this issue, the Transporter method was introduced for 2D data, which reconstructs the target frame from the source frame to incorporate both spatial and temporal information. However, the direct application of the Transporter to 3D point clouds is infeasible due to their structural differences from 2D images. Thus, we propose the first 3D version of the Transporter, which leverages hybrid 3D representation, cross attention, and implicit reconstruction. We apply this new learning system on 3D articulated objects and nonrigid animals (humans and rodents) and show that learned keypoints are spatio-temporally consistent. Additionally, we propose a closed-loop control strategy that utilizes the learned keypoints for 3D object manipulation and demonstrate its superior performance. Codes are available at https://github.com/zhongcl-thu/3D-Implicit-Transporter.