SpotLight: Shadow-Guided Object Relighting via Diffusion
This addresses the need for manual lighting control in neural rendering for applications like image editing and personalization, though it is incremental as it builds on existing neural rendering approaches.
The paper tackles the problem of precise lighting control in diffusion-based neural rendering for object relighting by using a coarse shadow as input, achieving superior object compositing results as confirmed by a user study.
Recent work has shown that diffusion models can serve as powerful neural rendering engines that can be leveraged for inserting virtual objects into images. However, unlike typical physics-based renderers, these neural rendering engines are limited by the lack of manual control over the lighting, which is often essential for improving or personalizing the desired image outcome. In this paper, we show that precise lighting control can be achieved for object relighting simply by providing a coarse shadow of the object. Indeed, we show that injecting only the desired shadow of the object into a pre-trained diffusion-based neural renderer enables it to accurately shade the object according to the desired light position, while properly harmonizing the object (and its shadow) within the target background image. Our method, SpotLight, leverages existing neural rendering approaches and achieves controllable relighting results with no additional training. We show that SpotLight achieves superior object compositing results, both quantitatively and perceptually, as confirmed by a user study, outperforming existing diffusion-based models specifically designed for relighting. We also demonstrate other applications, such as hand-scribbling shadows and full-image relighting, demonstrating its versatility.