Audio-Assisted Face Video Restoration with Temporal and Identity Complementary Learning
This addresses the challenge of enhancing face video quality in streaming applications, offering a general solution beyond prior audio-aided methods focused only on compression artifacts.
The paper tackles the problem of restoring degraded face videos by leveraging audio-visual correlations, proposing a network that integrates temporal and identity features with audio assistance to improve quality across compression artifacts, deblurring, and super-resolution, achieving state-of-the-art results.
Face videos accompanied by audio have become integral to our daily lives, while they often suffer from complex degradations. Most face video restoration methods neglect the intrinsic correlations between the visual and audio features, especially in mouth regions. A few audio-aided face video restoration methods have been proposed, but they only focus on compression artifact removal. In this paper, we propose a General Audio-assisted face Video restoration Network (GAVN) to address various types of streaming video distortions via identity and temporal complementary learning. Specifically, GAVN first captures inter-frame temporal features in the low-resolution space to restore frames coarsely and save computational cost. Then, GAVN extracts intra-frame identity features in the high-resolution space with the assistance of audio signals and face landmarks to restore more facial details. Finally, the reconstruction module integrates temporal features and identity features to generate high-quality face videos. Experimental results demonstrate that GAVN outperforms the existing state-of-the-art methods on face video compression artifact removal, deblurring, and super-resolution. Codes will be released upon publication.