GR3EN: Generative Relighting for 3D Environments
This addresses the challenge of high-quality 3D scene relighting for applications in computer graphics and vision, though it builds incrementally on existing generative methods.
The paper tackles the problem of relighting 3D reconstructions of large room-scale environments by distilling a video-to-video relighting diffusion model into a 3D reconstruction, avoiding difficult inverse rendering and enabling controllable relighting of complex real-world scenes.
We present a method for relighting 3D reconstructions of large room-scale environments. Existing solutions for 3D scene relighting often require solving under-determined or ill-conditioned inverse rendering problems, and are as such unable to produce high-quality results on complex real-world scenes. Though recent progress in using generative image and video diffusion models for relighting has been promising, these techniques are either limited to 2D image and video relighting or 3D relighting of individual objects. Our approach enables controllable 3D relighting of room-scale scenes by distilling the outputs of a video-to-video relighting diffusion model into a 3D reconstruction. This side-steps the need to solve a difficult inverse rendering problem, and results in a flexible system that can relight 3D reconstructions of complex real-world scenes. We validate our approach on both synthetic and real-world datasets to show that it can faithfully render novel views of scenes under new lighting conditions.