Towards Realistic Scene Generation with LiDAR Diffusion Models
This work addresses the problem of realistic 3D scene generation for autonomous driving and robotics, though it appears incremental as it adapts existing diffusion model techniques to a specific domain.
The paper tackles the challenge of generating realistic LiDAR scenes using diffusion models by proposing LiDAR Diffusion Models (LiDMs) that incorporate geometric priors, achieving competitive performance in unconditional generation and state-of-the-art results in conditional generation while being up to 107 times faster than point-based methods.
Diffusion models (DMs) excel in photo-realistic image synthesis, but their adaptation to LiDAR scene generation poses a substantial hurdle. This is primarily because DMs operating in the point space struggle to preserve the curve-like patterns and 3D geometry of LiDAR scenes, which consumes much of their representation power. In this paper, we propose LiDAR Diffusion Models (LiDMs) to generate LiDAR-realistic scenes from a latent space tailored to capture the realism of LiDAR scenes by incorporating geometric priors into the learning pipeline. Our method targets three major desiderata: pattern realism, geometry realism, and object realism. Specifically, we introduce curve-wise compression to simulate real-world LiDAR patterns, point-wise coordinate supervision to learn scene geometry, and patch-wise encoding for a full 3D object context. With these three core designs, our method achieves competitive performance on unconditional LiDAR generation in 64-beam scenario and state of the art on conditional LiDAR generation, while maintaining high efficiency compared to point-based DMs (up to 107$\times$ faster). Furthermore, by compressing LiDAR scenes into a latent space, we enable the controllability of DMs with various conditions such as semantic maps, camera views, and text prompts.