Learning to Aggregate and Personalize 3D Face from In-the-Wild Photo Collection
This addresses robust 3D face reconstruction for computer vision applications, but it is incremental as it builds on existing non-parametric modeling with novel components.
The paper tackles the problem of ambiguous noise and over-dependence on local color in non-parametric 3D face modeling from unconstrained photos, resulting in a method that recovers superior or competitive face shape and texture compared to state-of-the-art approaches.
Non-parametric face modeling aims to reconstruct 3D face only from images without shape assumptions. While plausible facial details are predicted, the models tend to over-depend on local color appearance and suffer from ambiguous noise. To address such problem, this paper presents a novel Learning to Aggregate and Personalize (LAP) framework for unsupervised robust 3D face modeling. Instead of using controlled environment, the proposed method implicitly disentangles ID-consistent and scene-specific face from unconstrained photo set. Specifically, to learn ID-consistent face, LAP adaptively aggregates intrinsic face factors of an identity based on a novel curriculum learning approach with relaxed consistency loss. To adapt the face for a personalized scene, we propose a novel attribute-refining network to modify ID-consistent face with target attribute and details. Based on the proposed method, we make unsupervised 3D face modeling benefit from meaningful image facial structure and possibly higher resolutions. Extensive experiments on benchmarks show LAP recovers superior or competitive face shape and texture, compared with state-of-the-art (SOTA) methods with or without prior and supervision.