SINGER: Vivid Audio-driven Singing Video Generation with Multi-scale Spectral Diffusion Model
This addresses the challenge of creating realistic singing videos for applications in entertainment or virtual avatars, but it is incremental as it builds on existing diffusion models with domain-specific adaptations.
The paper tackles the problem of generating singing videos from audio, which is underexplored compared to talking face generation, by proposing SINGER, a model that integrates multi-scale spectral and spectral-filtering modules into a diffusion framework, and it outperforms state-of-the-art methods in objective and subjective evaluations.
Recent advancements in generative models have significantly enhanced talking face video generation, yet singing video generation remains underexplored. The differences between human talking and singing limit the performance of existing talking face video generation models when applied to singing. The fundamental differences between talking and singing-specifically in audio characteristics and behavioral expressions-limit the effectiveness of existing models. We observe that the differences between singing and talking audios manifest in terms of frequency and amplitude. To address this, we have designed a multi-scale spectral module to help the model learn singing patterns in the spectral domain. Additionally, we develop a spectral-filtering module that aids the model in learning the human behaviors associated with singing audio. These two modules are integrated into the diffusion model to enhance singing video generation performance, resulting in our proposed model, SINGER. Furthermore, the lack of high-quality real-world singing face videos has hindered the development of the singing video generation community. To address this gap, we have collected an in-the-wild audio-visual singing dataset to facilitate research in this area. Our experiments demonstrate that SINGER is capable of generating vivid singing videos and outperforms state-of-the-art methods in both objective and subjective evaluations.