FaceCLIPNeRF: Text-driven 3D Face Manipulation using Deformable Neural Radiance Fields
This addresses the challenge of making 3D face manipulation accessible to non-expert users by reducing labor-intensive requirements, though it is incremental as it builds on existing NeRF and CLIP technologies.
The paper tackles the problem of manipulating 3D faces reconstructed with Neural Radiance Fields (NeRF) by enabling text-driven control, eliminating the need for manual input like semantic masks or attribute searches. It achieves this by training a deformable NeRF with a Position-conditional Anchor Compositor to optimize renderings for high similarity to target text in CLIP embedding space, demonstrating effectiveness through extensive results.
As recent advances in Neural Radiance Fields (NeRF) have enabled high-fidelity 3D face reconstruction and novel view synthesis, its manipulation also became an essential task in 3D vision. However, existing manipulation methods require extensive human labor, such as a user-provided semantic mask and manual attribute search unsuitable for non-expert users. Instead, our approach is designed to require a single text to manipulate a face reconstructed with NeRF. To do so, we first train a scene manipulator, a latent code-conditional deformable NeRF, over a dynamic scene to control a face deformation using the latent code. However, representing a scene deformation with a single latent code is unfavorable for compositing local deformations observed in different instances. As so, our proposed Position-conditional Anchor Compositor (PAC) learns to represent a manipulated scene with spatially varying latent codes. Their renderings with the scene manipulator are then optimized to yield high cosine similarity to a target text in CLIP embedding space for text-driven manipulation. To the best of our knowledge, our approach is the first to address the text-driven manipulation of a face reconstructed with NeRF. Extensive results, comparisons, and ablation studies demonstrate the effectiveness of our approach.