Lighting in Motion: Spatiotemporal HDR Lighting Estimation
This work addresses the problem of realistic lighting simulation for computer graphics and vision applications, representing an incremental improvement with novel conditioning techniques.
The paper tackles spatiotemporal lighting estimation by proposing LiMo, a diffusion-based method that generates high-dynamic-range illumination maps from input scenes, achieving state-of-the-art results in spatial control and prediction accuracy.
We present Lighting in Motion (LiMo), a diffusion-based approach to spatiotemporal lighting estimation. LiMo targets both realistic high-frequency detail prediction and accurate illuminance estimation. To account for both, we propose generating a set of mirrored and diffuse spheres at different exposures, based on their 3D positions in the input. Making use of diffusion priors, we fine-tune powerful existing diffusion models on a large-scale customized dataset of indoor and outdoor scenes, paired with spatiotemporal light probes. For accurate spatial conditioning, we demonstrate that depth alone is insufficient and we introduce a new geometric condition to provide the relative position of the scene to the target 3D position. Finally, we combine diffuse and mirror predictions at different exposures into a single HDRI map leveraging differentiable rendering. We thoroughly evaluate our method and design choices to establish LiMo as state-of-the-art for both spatial control and prediction accuracy.