Real-time Simultaneous 3D Head Modeling and Facial Motion Capture with an RGB-D camera
This enables instant, real-time creation of animated 3D head models for applications like gaming or VR, representing a novel integration rather than an incremental improvement.
The authors tackled the problem of real-time 3D head modeling and facial motion capture using a consumer RGB-D camera, achieving simultaneous tracking and modeling without training or pre-scanning, with robust results across various subjects and expressions.
We propose a method to build in real-time animated 3D head models using a consumer-grade RGB-D camera. Our proposed method is the first one to provide simultaneously comprehensive facial motion tracking and a detailed 3D model of the user's head. Anyone's head can be instantly reconstructed and his facial motion captured without requiring any training or pre-scanning. The user starts facing the camera with a neutral expression in the first frame, but is free to move, talk and change his face expression as he wills otherwise. The facial motion is captured using a blendshape animation model while geometric details are captured using a Deviation image mapped over the template mesh. We contribute with an efficient algorithm to grow and refine the deforming 3D model of the head on-the-fly and in real-time. We demonstrate robust and high-fidelity simultaneous facial motion capture and 3D head modeling results on a wide range of subjects with various head poses and facial expressions.