Towards Accurate Alignment in Real-time 3D Hand-Mesh Reconstruction
This work addresses the problem of enabling realistic hand interactions in augmented reality applications, representing an incremental improvement over existing methods.
The paper tackled the challenge of achieving real-time 3D hand-mesh reconstruction from RGB images with accurate pose, shape, and mesh-image alignment, resulting in a method that outperforms state-of-the-art on precision and alignment benchmarks.
3D hand-mesh reconstruction from RGB images facilitates many applications, including augmented reality (AR). However, this requires not only real-time speed and accurate hand pose and shape but also plausible mesh-image alignment. While existing works already achieve promising results, meeting all three requirements is very challenging. This paper presents a novel pipeline by decoupling the hand-mesh reconstruction task into three stages: a joint stage to predict hand joints and segmentation; a mesh stage to predict a rough hand mesh; and a refine stage to fine-tune it with an offset mesh for mesh-image alignment. With careful design in the network structure and in the loss functions, we can promote high-quality finger-level mesh-image alignment and drive the models together to deliver real-time predictions. Extensive quantitative and qualitative results on benchmark datasets demonstrate that the quality of our results outperforms the state-of-the-art methods on hand-mesh/pose precision and hand-image alignment. In the end, we also showcase several real-time AR scenarios.