Generative Adversarial Talking Head: Bringing Portraits to Life with a Weakly Supervised Neural Network
This enables template-and-target-free expression editing for still faces, benefiting applications in digital media and entertainment, though it is incremental as it builds on prior generative adversarial networks.
The paper tackles the problem of automatically synthesizing facial expressions in arbitrary portrait photos using continuous action unit coefficients, achieving a method that maintains personal characteristics and background without requiring statistical face models or image processing tricks.
This paper presents Generative Adversarial Talking Head (GATH), a novel deep generative neural network that enables fully automatic facial expression synthesis of an arbitrary portrait with continuous action unit (AU) coefficients. Specifically, our model directly manipulates image pixels to make the unseen subject in the still photo express various emotions controlled by values of facial AU coefficients, while maintaining her personal characteristics, such as facial geometry, skin color and hair style, as well as the original surrounding background. In contrast to prior work, GATH is purely data-driven and it requires neither a statistical face model nor image processing tricks to enact facial deformations. Additionally, our model is trained from unpaired data, where the input image, with its auxiliary identity label taken from abundance of still photos in the wild, and the target frame are from different persons. In order to effectively learn such model, we propose a novel weakly supervised adversarial learning framework that consists of a generator, a discriminator, a classifier and an action unit estimator. Our work gives rise to template-and-target-free expression editing, where still faces can be effortlessly animated with arbitrary AU coefficients provided by the user.