Generalizing Neural Human Fitting to Unseen Poses With Articulated SE(3) Equivariance
This work addresses the challenge of robust human body model fitting for computer vision applications, offering a novel solution that improves generalization and efficiency over existing methods.
The paper tackles the problem of fitting a parametric human body model (SMPL) to point cloud data by proposing ArtEq, a part-based SE(3)-equivariant neural architecture that generalizes to unseen poses, achieving a ~44% improvement in body reconstruction accuracy and being three orders of magnitude faster with 97.3% fewer parameters than prior methods.
We address the problem of fitting a parametric human body model (SMPL) to point cloud data. Optimization-based methods require careful initialization and are prone to becoming trapped in local optima. Learning-based methods address this but do not generalize well when the input pose is far from those seen during training. For rigid point clouds, remarkable generalization has been achieved by leveraging SE(3)-equivariant networks, but these methods do not work on articulated objects. In this work we extend this idea to human bodies and propose ArtEq, a novel part-based SE(3)-equivariant neural architecture for SMPL model estimation from point clouds. Specifically, we learn a part detection network by leveraging local SO(3) invariance, and regress shape and pose using articulated SE(3) shape-invariant and pose-equivariant networks, all trained end-to-end. Our novel pose regression module leverages the permutation-equivariant property of self-attention layers to preserve rotational equivariance. Experimental results show that ArtEq generalizes to poses not seen during training, outperforming state-of-the-art methods by ~44% in terms of body reconstruction accuracy, without requiring an optimization refinement step. Furthermore, ArtEq is three orders of magnitude faster during inference than prior work and has 97.3% fewer parameters. The code and model are available for research purposes at https://arteq.is.tue.mpg.de.