AVFace: Towards Detailed Audio-Visual 4D Face Reconstruction
This work addresses the challenge of detailed 4D face reconstruction for applications like AR/VR, where videos include both visual and speech data, but it is incremental as it builds on existing multimodal approaches.
The paper tackles the problem of 4D face reconstruction from monocular videos by proposing AVFace, a multimodal method that uses both audio and visual information to accurately reconstruct facial and lip motion without requiring 3D ground truth for training, achieving superior results over state-of-the-art methods in qualitative and quantitative evaluations.
In this work, we present a multimodal solution to the problem of 4D face reconstruction from monocular videos. 3D face reconstruction from 2D images is an under-constrained problem due to the ambiguity of depth. State-of-the-art methods try to solve this problem by leveraging visual information from a single image or video, whereas 3D mesh animation approaches rely more on audio. However, in most cases (e.g. AR/VR applications), videos include both visual and speech information. We propose AVFace that incorporates both modalities and accurately reconstructs the 4D facial and lip motion of any speaker, without requiring any 3D ground truth for training. A coarse stage estimates the per-frame parameters of a 3D morphable model, followed by a lip refinement, and then a fine stage recovers facial geometric details. Due to the temporal audio and video information captured by transformer-based modules, our method is robust in cases when either modality is insufficient (e.g. face occlusions). Extensive qualitative and quantitative evaluation demonstrates the superiority of our method over the current state-of-the-art.