AE-NeRF: Audio Enhanced Neural Radiance Field for Few Shot Talking Head Synthesis
This work improves audio-driven talking head generation for applications in digital humans, film, and VR, though it appears incremental by refining existing NeRF frameworks.
The paper tackles few-shot talking head synthesis by addressing limitations in existing NeRF-based approaches that lack audio-aware priors and ignore regional audio correlations, resulting in AE-NeRF which surpasses state-of-the-art methods in image fidelity, audio-lip synchronization, and generalization with limited data.
Audio-driven talking head synthesis is a promising topic with wide applications in digital human, film making and virtual reality. Recent NeRF-based approaches have shown superiority in quality and fidelity compared to previous studies. However, when it comes to few-shot talking head generation, a practical scenario where only few seconds of talking video is available for one identity, two limitations emerge: 1) they either have no base model, which serves as a facial prior for fast convergence, or ignore the importance of audio when building the prior; 2) most of them overlook the degree of correlation between different face regions and audio, e.g., mouth is audio related, while ear is audio independent. In this paper, we present Audio Enhanced Neural Radiance Field (AE-NeRF) to tackle the above issues, which can generate realistic portraits of a new speaker with fewshot dataset. Specifically, we introduce an Audio Aware Aggregation module into the feature fusion stage of the reference scheme, where the weight is determined by the similarity of audio between reference and target image. Then, an Audio-Aligned Face Generation strategy is proposed to model the audio related and audio independent regions respectively, with a dual-NeRF framework. Extensive experiments have shown AE-NeRF surpasses the state-of-the-art on image fidelity, audio-lip synchronization, and generalization ability, even in limited training set or training iterations.