ESARM: 3D Emotional Speech-to-Animation via Reward Model from Automatically-Ranked Demonstrations
This addresses the need for more expressive and varied facial animations in applications like virtual avatars or entertainment, though it appears incremental by building on existing STA models.
The paper tackles the problem of generating 3D speech-to-animation with emotional depth and diversity, achieving high-quality, emotionally rich animations that better align with human preferences.
This paper proposes a novel 3D speech-to-animation (STA) generation framework designed to address the shortcomings of existing models in producing diverse and emotionally resonant animations. Current STA models often generate animations that lack emotional depth and variety, failing to align with human expectations. To overcome these limitations, we introduce a novel STA model coupled with a reward model. This combination enables the decoupling of emotion and content under audio conditions through a cross-coupling training approach. Additionally, we develop a training methodology that leverages automatic quality evaluation of generated facial animations to guide the reinforcement learning process. This methodology encourages the STA model to explore a broader range of possibilities, resulting in the generation of diverse and emotionally expressive facial animations of superior quality. We conduct extensive empirical experiments on a benchmark dataset, and the results validate the effectiveness of our proposed framework in generating high-quality, emotionally rich 3D animations that are better aligned with human preferences.