Weakly-supervised Single-view Image Relighting
This work addresses the challenge of realistically inserting objects from photos into new scenes under different lighting for AR, which is incremental by improving on existing methods with a novel dataset and constraints.
The paper tackles the problem of relighting single images of objects for AR applications by proposing a weakly-supervised learning approach that solves inverse rendering and re-rendering, achieving state-of-the-art performance as demonstrated in extensive evaluations.
We present a learning-based approach to relight a single image of Lambertian and low-frequency specular objects. Our method enables inserting objects from photographs into new scenes and relighting them under the new environment lighting, which is essential for AR applications. To relight the object, we solve both inverse rendering and re-rendering. To resolve the ill-posed inverse rendering, we propose a weakly-supervised method by a low-rank constraint. To facilitate the weakly-supervised training, we contribute Relit, a large-scale (750K images) dataset of videos with aligned objects under changing illuminations. For re-rendering, we propose a differentiable specular rendering layer to render low-frequency non-Lambertian materials under various illuminations of spherical harmonics. The whole pipeline is end-to-end and efficient, allowing for a mobile app implementation of AR object insertion. Extensive evaluations demonstrate that our method achieves state-of-the-art performance. Project page: https://renjiaoyi.github.io/relighting/.