RealisticHands: A Hybrid Model for 3D Hand Reconstruction
This work addresses the problem of accurate 3D hand reconstruction for VR applications, representing an incremental improvement over existing methods.
The paper tackles robust 3D hand mesh estimation from RGB images by proposing a hybrid method combining deep neural networks and differential rendering optimization, achieving improved image-model alignment and demonstrating application in VR for realistic hand texture acquisition.
Estimating 3D hand meshes from RGB images robustly is a highly desirable task, made challenging due to the numerous degrees of freedom, and issues such as self similarity and occlusions. Previous methods generally either use parametric 3D hand models or follow a model-free approach. While the former can be considered more robust, e.g. to occlusions, they are less expressive. We propose a hybrid approach, utilizing a deep neural network and differential rendering based optimization to demonstrably achieve the best of both worlds. In addition, we explore Virtual Reality (VR) as an application. Most VR headsets are nowadays equipped with multiple cameras, which we can leverage by extending our method to the egocentric stereo domain. This extension proves to be more resilient to the above mentioned issues. Finally, as a use-case, we show that the improved image-model alignment can be used to acquire the user's hand texture, which leads to a more realistic virtual hand representation.