Perspective Reconstruction of Human Faces by Joint Mesh and Landmark Regression
This addresses a specific issue in computer vision for applications like augmented reality, though it is incremental as it builds on existing orthogonal projection methods.
The paper tackles the problem of inaccurate 3D face reconstruction under perspective projection when faces are close to the camera by jointly regressing 3D mesh and 2D landmarks, achieving first place in the ECCV 2022 WCPA challenge.
Even though 3D face reconstruction has achieved impressive progress, most orthogonal projection-based face reconstruction methods can not achieve accurate and consistent reconstruction results when the face is very close to the camera due to the distortion under the perspective projection. In this paper, we propose to simultaneously reconstruct 3D face mesh in the world space and predict 2D face landmarks on the image plane to address the problem of perspective 3D face reconstruction. Based on the predicted 3D vertices and 2D landmarks, the 6DoF (6 Degrees of Freedom) face pose can be easily estimated by the PnP solver to represent perspective projection. Our approach achieves 1st place on the leader-board of the ECCV 2022 WCPA challenge and our model is visually robust under different identities, expressions and poses. The training code and models are released to facilitate future research.