Reconstructing Hands in 3D with Transformers
This addresses the problem of accurate 3D hand modeling for applications like VR/AR and human-computer interaction, representing an incremental improvement through scaling data and model capacity.
The paper tackles 3D hand reconstruction from monocular input using a transformer-based architecture called HaMeR, achieving significantly increased accuracy and robustness over previous work, with consistent outperformance on popular benchmarks and a new in-the-wild dataset.
We present an approach that can reconstruct hands in 3D from monocular input. Our approach for Hand Mesh Recovery, HaMeR, follows a fully transformer-based architecture and can analyze hands with significantly increased accuracy and robustness compared to previous work. The key to HaMeR's success lies in scaling up both the data used for training and the capacity of the deep network for hand reconstruction. For training data, we combine multiple datasets that contain 2D or 3D hand annotations. For the deep model, we use a large scale Vision Transformer architecture. Our final model consistently outperforms the previous baselines on popular 3D hand pose benchmarks. To further evaluate the effect of our design in non-controlled settings, we annotate existing in-the-wild datasets with 2D hand keypoint annotations. On this newly collected dataset of annotations, HInt, we demonstrate significant improvements over existing baselines. We make our code, data and models available on the project website: https://geopavlakos.github.io/hamer/.