RPM-Net: Recurrent Prediction of Motion and Parts from Point Cloud
This addresses the challenge of understanding object structure and dynamics from incomplete 3D data, which is incremental as it builds on existing deep learning methods for point clouds.
The paper tackles the problem of simultaneously inferring movable parts and predicting their motions from a single, unsegmented, and possibly partial 3D point cloud, using RPM-Net, a recurrent neural network that achieves motion-based shape segmentation and hierarchical object segmentation.
We introduce RPM-Net, a deep learning-based approach which simultaneously infers movable parts and hallucinates their motions from a single, un-segmented, and possibly partial, 3D point cloud shape. RPM-Net is a novel Recurrent Neural Network (RNN), composed of an encoder-decoder pair with interleaved Long Short-Term Memory (LSTM) components, which together predict a temporal sequence of pointwise displacements for the input point cloud. At the same time, the displacements allow the network to learn movable parts, resulting in a motion-based shape segmentation. Recursive applications of RPM-Net on the obtained parts can predict finer-level part motions, resulting in a hierarchical object segmentation. Furthermore, we develop a separate network to estimate part mobilities, e.g., per-part motion parameters, from the segmented motion sequence. Both networks learn deep predictive models from a training set that exemplifies a variety of mobilities for diverse objects. We show results of simultaneous motion and part predictions from synthetic and real scans of 3D objects exhibiting a variety of part mobilities, possibly involving multiple movable parts.