Robust Human Registration with Body Part Segmentation on Noisy Point Clouds
This work addresses the challenge of precise human registration for applications such as augmented reality and human-robot interaction, representing an incremental advancement over prior methods.
The paper tackles the problem of registering human meshes to noisy 3D point clouds by incorporating body-part segmentation into the fitting process, resulting in significant improvements in both pose estimation and segmentation accuracy on real-world datasets like InterCap, EgoBody, and BEHAVE.
Registering human meshes to 3D point clouds is essential for applications such as augmented reality and human-robot interaction but often yields imprecise results due to noise and background clutter in real-world data. We introduce a hybrid approach that incorporates body-part segmentation into the mesh fitting process, enhancing both human pose estimation and segmentation accuracy. Our method first assigns body part labels to individual points, which then guide a two-step SMPL-X fitting: initial pose and orientation estimation using body part centroids, followed by global refinement of the point cloud alignment. Additionally, we demonstrate that the fitted human mesh can refine body part labels, leading to improved segmentation. Evaluations on the cluttered and noisy real-world datasets InterCap, EgoBody, and BEHAVE show that our approach significantly outperforms prior methods in both pose estimation and segmentation accuracy. Code and results are available on our project website: https://segfit.github.io