Learning Physics-guided Face Relighting under Directional Light
This work addresses the need for authentic face relighting in applications like augmented reality telepresence, representing a novel method for a known bottleneck in computer vision.
The paper tackled the problem of realistically relighting human faces in images by developing an end-to-end deep learning model that decomposes images into intrinsic components and predicts residual corrections for non-diffuse effects, achieving precise and believable results that generalize to complex illumination and challenging poses.
Relighting is an essential step in realistically transferring objects from a captured image into another environment. For example, authentic telepresence in Augmented Reality requires faces to be displayed and relit consistent with the observer's scene lighting. We investigate end-to-end deep learning architectures that both de-light and relight an image of a human face. Our model decomposes the input image into intrinsic components according to a diffuse physics-based image formation model. We enable non-diffuse effects including cast shadows and specular highlights by predicting a residual correction to the diffuse render. To train and evaluate our model, we collected a portrait database of 21 subjects with various expressions and poses. Each sample is captured in a controlled light stage setup with 32 individual light sources. Our method creates precise and believable relighting results and generalizes to complex illumination conditions and challenging poses, including when the subject is not looking straight at the camera.