SingingHead: A Large-scale 4D Dataset for Singing Head Animation
This addresses the problem of audio-driven facial animation for singing, which is overlooked due to domain gaps, by providing a dataset and method for researchers and developers in entertainment and communication.
The authors tackled the lack of datasets for singing head animation by collecting SingingHead, a large-scale dataset with over 27 hours of synchronized data from 76 individuals, and proposed UniSinger, a unified framework that achieves competitive results on 3D and 2D benchmarks.
Singing, as a common facial movement second only to talking, can be regarded as a universal language across ethnicities and cultures, plays an important role in emotional communication, art, and entertainment. However, it is often overlooked in the field of audio-driven facial animation due to the lack of singing head datasets and the domain gap between singing and talking in rhythm and amplitude. To this end, we collect a high-quality large-scale singing head dataset, SingingHead, which consists of more than 27 hours of synchronized singing video, 3D facial motion, singing audio, and background music from 76 individuals and 8 types of music. Along with the SingingHead dataset, we benchmark existing audio-driven 3D facial animation methods and 2D talking head methods on the singing task. Furthermore, we argue that 3D and 2D facial animation tasks can be solved together, and propose a unified singing head animation framework named UniSinger to achieve both singing audio-driven 3D singing head animation and 2D singing portrait video synthesis, which achieves competitive results on both 3D and 2D benchmarks. Extensive experiments demonstrate the significance of the proposed singing-specific dataset in promoting the development of singing head animation tasks, as well as the promising performance of our unified facial animation framework.