LightSim: Neural Lighting Simulation for Urban Scenes
This addresses the need for cost-effective, diverse training data for image-based robot perception systems in urban environments, though it is incremental as it builds on prior simulation and rendering methods.
The paper tackles the problem of outdoor illumination variations harming robot perception by proposing LightSim, a neural lighting camera simulation system that generates realistic, controllable urban scene images under different lighting conditions, and shows that training perception models on this data significantly improves performance.
Different outdoor illumination conditions drastically alter the appearance of urban scenes, and they can harm the performance of image-based robot perception systems if not seen during training. Camera simulation provides a cost-effective solution to create a large dataset of images captured under different lighting conditions. Towards this goal, we propose LightSim, a neural lighting camera simulation system that enables diverse, realistic, and controllable data generation. LightSim automatically builds lighting-aware digital twins at scale from collected raw sensor data and decomposes the scene into dynamic actors and static background with accurate geometry, appearance, and estimated scene lighting. These digital twins enable actor insertion, modification, removal, and rendering from a new viewpoint, all in a lighting-aware manner. LightSim then combines physically-based and learnable deferred rendering to perform realistic relighting of modified scenes, such as altering the sun location and modifying the shadows or changing the sun brightness, producing spatially- and temporally-consistent camera videos. Our experiments show that LightSim generates more realistic relighting results than prior work. Importantly, training perception models on data generated by LightSim can significantly improve their performance.