SfSNet: Learning Shape, Reflectance and Illuminance of Faces in the Wild
This addresses the challenge of accurate inverse rendering for faces in the wild, which is important for applications in computer vision and graphics, though it appears incremental as it builds on existing physical models and methods.
The paper tackles the problem of decomposing unconstrained human face images into shape, reflectance, and illuminance using SfSNet, an end-to-end learning framework that combines synthetic and real-world data, resulting in significantly better quantitative and qualitative outcomes than state-of-the-art methods.
We present SfSNet, an end-to-end learning framework for producing an accurate decomposition of an unconstrained human face image into shape, reflectance and illuminance. SfSNet is designed to reflect a physical lambertian rendering model. SfSNet learns from a mixture of labeled synthetic and unlabeled real world images. This allows the network to capture low frequency variations from synthetic and high frequency details from real images through the photometric reconstruction loss. SfSNet consists of a new decomposition architecture with residual blocks that learns a complete separation of albedo and normal. This is used along with the original image to predict lighting. SfSNet produces significantly better quantitative and qualitative results than state-of-the-art methods for inverse rendering and independent normal and illumination estimation.