Towards a Simultaneous and Granular Identity-Expression Control in Personalized Face Generation
This addresses the challenge for users in human-centric content generation who need precise control over identity and expression in portrait images, representing a novel integration rather than an incremental improvement.
The paper tackles the problem of generating personalized face images that retain a specific individual's identity while exhibiting diverse, fine-grained expressions, achieving state-of-the-art performance in simultaneous identity-expression control compared to existing methods.
In human-centric content generation, the pre-trained text-to-image models struggle to produce user-wanted portrait images, which retain the identity of individuals while exhibiting diverse expressions. This paper introduces our efforts towards personalized face generation. To this end, we propose a novel multi-modal face generation framework, capable of simultaneous identity-expression control and more fine-grained expression synthesis. Our expression control is so sophisticated that it can be specialized by the fine-grained emotional vocabulary. We devise a novel diffusion model that can undertake the task of simultaneously face swapping and reenactment. Due to the entanglement of identity and expression, it's nontrivial to separately and precisely control them in one framework, thus has not been explored yet. To overcome this, we propose several innovative designs in the conditional diffusion model, including balancing identity and expression encoder, improved midpoint sampling, and explicitly background conditioning. Extensive experiments have demonstrated the controllability and scalability of the proposed framework, in comparison with state-of-the-art text-to-image, face swapping, and face reenactment methods.