Face De-occlusion using 3D Morphable Model and Generative Adversarial Network
This addresses face reconstruction issues in computer vision for applications like security or entertainment, but it is incremental as it builds on existing 3DMM and GAN techniques.
The paper tackles the problem of face occlusion by objects like eyeglasses and masks, which hinders 3D face reconstruction, and proposes a method using 3D morphable models and generative adversarial networks to restore de-occluded face images, achieving effective removal of occlusions and correct 3D model reconstruction.
In recent decades, 3D morphable model (3DMM) has been commonly used in image-based photorealistic 3D face reconstruction. However, face images are often corrupted by serious occlusion by non-face objects including eyeglasses, masks, and hands. Such objects block the correct capture of landmarks and shading information. Therefore, the reconstructed 3D face model is hardly reusable. In this paper, a novel method is proposed to restore de-occluded face images based on inverse use of 3DMM and generative adversarial network. We utilize the 3DMM prior to the proposed adversarial network and combine a global and local adversarial convolutional neural network to learn face de-occlusion model. The 3DMM serves not only as geometric prior but also proposes the face region for the local discriminator. Experiment results confirm the effectiveness and robustness of the proposed algorithm in removing challenging types of occlusions with various head poses and illumination. Furthermore, the proposed method reconstructs the correct 3D face model with de-occluded textures.