XFormer: Fast and Accurate Monocular 3D Body Capture
This enables fast and accurate 3D body capture from single images, beneficial for applications like AR/VR and animation, though it is incremental as it builds on existing transformer and backbone architectures.
The authors tackled real-time monocular 3D human mesh and motion capture by proposing XFormer, a method that uses a cross-modal transformer to integrate 2D keypoint and image features, achieving over 30fps on a CPU core and state-of-the-art accuracy on Huamn3.6 and 3DPW datasets.
We present XFormer, a novel human mesh and motion capture method that achieves real-time performance on consumer CPUs given only monocular images as input. The proposed network architecture contains two branches: a keypoint branch that estimates 3D human mesh vertices given 2D keypoints, and an image branch that makes predictions directly from the RGB image features. At the core of our method is a cross-modal transformer block that allows information to flow across these two branches by modeling the attention between 2D keypoint coordinates and image spatial features. Our architecture is smartly designed, which enables us to train on various types of datasets including images with 2D/3D annotations, images with 3D pseudo labels, and motion capture datasets that do not have associated images. This effectively improves the accuracy and generalization ability of our system. Built on a lightweight backbone (MobileNetV3), our method runs blazing fast (over 30fps on a single CPU core) and still yields competitive accuracy. Furthermore, with an HRNet backbone, XFormer delivers state-of-the-art performance on Huamn3.6 and 3DPW datasets.