MoCap-to-Visual Domain Adaptation for Efficient Human Mesh Estimation from 2D Keypoints
This work addresses the scarcity of labeled image data for 3D human mesh estimation, enabling efficient and accurate applications in computer vision without costly annotations.
The paper tackles the problem of estimating 3D human meshes from 2D keypoints without using RGB images or paired mesh labels, by training on motion capture data and applying adversarial domain adaptation to bridge the gap to visual domains. The results show that Key2Mesh sets a new state-of-the-art, outperforming other models in PA-MPJPE on H3.6M and 3DPW datasets and in MPJPE and PVE on 3DPW, while operating at least 12x faster than prior methods.
This paper presents Key2Mesh, a model that takes a set of 2D human pose keypoints as input and estimates the corresponding body mesh. Since this process does not involve any visual (i.e. RGB image) data, the model can be trained on large-scale motion capture (MoCap) datasets, thereby overcoming the scarcity of image datasets with 3D labels. To enable the model's application on RGB images, we first run an off-the-shelf 2D pose estimator to obtain the 2D keypoints, and then feed these 2D keypoints to Key2Mesh. To improve the performance of our model on RGB images, we apply an adversarial domain adaptation (DA) method to bridge the gap between the MoCap and visual domains. Crucially, our DA method does not require 3D labels for visual data, which enables adaptation to target sets without the need for costly labels. We evaluate Key2Mesh for the task of estimating 3D human meshes from 2D keypoints, in the absence of RGB and mesh label pairs. Our results on widely used H3.6M and 3DPW datasets show that Key2Mesh sets the new state-of-the-art by outperforming other models in PA-MPJPE for both datasets, and in MPJPE and PVE for the 3DPW dataset. Thanks to our model's simple architecture, it operates at least 12x faster than the prior state-of-the-art model, LGD. Additional qualitative samples and code are available on the project website: https://key2mesh.github.io/.