IMU2Face: Real-time Gesture-driven Facial Reenactment
This addresses the problem of intuitive facial animation for users in entertainment or communication, but it is incremental as it builds on existing Face2Face technology.
The paper tackles real-time facial reenactment by using hand gestures tracked with IMUs to control facial expressions in a target video, achieving real-time performance without specifying concrete numbers.
We present IMU2Face, a gesture-driven facial reenactment system. To this end, we combine recent advances in facial motion capture and inertial measurement units (IMUs) to control the facial expressions of a person in a target video based on intuitive hand gestures. IMUs are omnipresent, since modern smart-phones, smart-watches and drones integrate such sensors, e.g., for changing the orientation of the screen content, counting steps, or for flight stabilization. Face tracking and reenactment is based on the state-of-the-art real-time Face2Face facial reenactment system. Instead of transferring facial expressions from a source to a target actor, we employ an IMU to track the hand gestures of a source actor and use its orientation to modify the target actor's expressions.