Robust Point Light Source Estimation Using Differentiable Rendering
This work addresses illumination estimation for mixed reality, enabling consistent virtual object insertion, but appears incremental as it builds on existing differentiable rendering and Blinn-Phong models.
The paper tackles the problem of estimating point light sources from RGBD images for mixed reality applications by formulating it as an inverse problem and proposing a novel differentiable renderer based on the Blinn-Phong model with cast shadows, demonstrating robustness to incorrect reflectance estimation.
Illumination estimation is often used in mixed reality to re-render a scene from another point of view, to change the color/texture of an object, or to insert a virtual object consistently lit into a real video or photograph. Specifically, the estimation of a point light source is required for the shadows cast by the inserted object to be consistent with the real scene. We tackle the problem of illumination retrieval given an RGBD image of the scene as an inverse problem: we aim to find the illumination that minimizes the photometric error between the rendered image and the observation. In particular we propose a novel differentiable renderer based on the Blinn-Phong model with cast shadows. We compare our differentiable renderer to state-of-the-art methods and demonstrate its robustness to an incorrect reflectance estimation.