Multiple View Geometry Transformers for 3D Human Pose Estimation
This addresses occlusion and generalization issues in 3D human pose estimation for computer vision applications, representing a novel hybrid approach rather than a purely incremental improvement.
The paper tackles the problem of 3D human pose estimation from multiple views by improving 3D reasoning in Transformers, proposing MVGFormer, a hybrid model with learning-free geometry and learnable appearance modules that outperforms state-of-the-art methods, especially in out-of-domain settings.
In this work, we aim to improve the 3D reasoning ability of Transformers in multi-view 3D human pose estimation. Recent works have focused on end-to-end learning-based transformer designs, which struggle to resolve geometric information accurately, particularly during occlusion. Instead, we propose a novel hybrid model, MVGFormer, which has a series of geometric and appearance modules organized in an iterative manner. The geometry modules are learning-free and handle all viewpoint-dependent 3D tasks geometrically which notably improves the model's generalization ability. The appearance modules are learnable and are dedicated to estimating 2D poses from image signals end-to-end which enables them to achieve accurate estimates even when occlusion occurs, leading to a model that is both accurate and generalizable to new cameras and geometries. We evaluate our approach for both in-domain and out-of-domain settings, where our model consistently outperforms state-of-the-art methods, and especially does so by a significant margin in the out-of-domain setting. We will release the code and models: https://github.com/XunshanMan/MVGFormer.