ControlFace: Harnessing Facial Parametric Control for Face Rigging
This addresses the challenge of retaining fine-grained identity and semantic details in face rigging for computer vision applications, representing a novel method for a known bottleneck rather than a foundational advancement.
The paper tackles the problem of face rigging, which involves manipulating facial images for specific controls like pose and expression, by introducing ControlFace, a method that uses 3DMM renderings to achieve flexible, high-fidelity control, resulting in superior performance in identity preservation and control precision as demonstrated in experiments.
Manipulation of facial images to meet specific controls such as pose, expression, and lighting, also known as face rigging, is a complex task in computer vision. Existing methods are limited by their reliance on image datasets, which necessitates individual-specific fine-tuning and limits their ability to retain fine-grained identity and semantic details, reducing practical usability. To overcome these limitations, we introduce ControlFace, a novel face rigging method conditioned on 3DMM renderings that enables flexible, high-fidelity control. We employ a dual-branch U-Nets: one, referred to as FaceNet, captures identity and fine details, while the other focuses on generation. To enhance control precision, the control mixer module encodes the correlated features between the target-aligned control and reference-aligned control, and a novel guidance method, reference control guidance, steers the generation process for better control adherence. By training on a facial video dataset, we fully utilize FaceNet's rich representations while ensuring control adherence. Extensive experiments demonstrate ControlFace's superior performance in identity preservation and control precision, highlighting its practicality. Please see the project website: https://cvlab-kaist.github.io/ControlFace/.