LumiNet: Latent Intrinsics Meets Diffusion Models for Indoor Scene Relighting
This addresses the problem of realistic lighting transfer in indoor scenes for computer vision and graphics applications, representing an incremental improvement over prior generative models.
The paper tackles the problem of indoor scene relighting by introducing LumiNet, which synthesizes relit images from source and target lighting inputs, outperforming existing methods on challenging indoor scenes with complex lighting effects like specular highlights and indirect illumination.
We introduce LumiNet, a novel architecture that leverages generative models and latent intrinsic representations for effective lighting transfer. Given a source image and a target lighting image, LumiNet synthesizes a relit version of the source scene that captures the target's lighting. Our approach makes two key contributions: a data curation strategy from the StyleGAN-based relighting model for our training, and a modified diffusion-based ControlNet that processes both latent intrinsic properties from the source image and latent extrinsic properties from the target image. We further improve lighting transfer through a learned adaptor (MLP) that injects the target's latent extrinsic properties via cross-attention and fine-tuning. Unlike traditional ControlNet, which generates images with conditional maps from a single scene, LumiNet processes latent representations from two different images - preserving geometry and albedo from the source while transferring lighting characteristics from the target. Experiments demonstrate that our method successfully transfers complex lighting phenomena including specular highlights and indirect illumination across scenes with varying spatial layouts and materials, outperforming existing approaches on challenging indoor scenes using only images as input.