LiDARDraft: Generating LiDAR Point Cloud from Versatile Inputs
This addresses the need for controllable and high-quality LiDAR data generation in autonomous driving simulation, enabling simulation from scratch from arbitrary inputs.
The paper tackles the problem of generating realistic and diverse LiDAR point clouds for autonomous driving simulation by proposing LiDARDraft, which uses 3D layouts to bridge versatile inputs like text and images with point clouds, achieving high-quality results with pixel-level alignment.
Generating realistic and diverse LiDAR point clouds is crucial for autonomous driving simulation. Although previous methods achieve LiDAR point cloud generation from user inputs, they struggle to attain high-quality results while enabling versatile controllability, due to the imbalance between the complex distribution of LiDAR point clouds and the simple control signals. To address the limitation, we propose LiDARDraft, which utilizes the 3D layout to build a bridge between versatile conditional signals and LiDAR point clouds. The 3D layout can be trivially generated from various user inputs such as textual descriptions and images. Specifically, we represent text, images, and point clouds as unified 3D layouts, which are further transformed into semantic and depth control signals. Then, we employ a rangemap-based ControlNet to guide LiDAR point cloud generation. This pixel-level alignment approach demonstrates excellent performance in controllable LiDAR point clouds generation, enabling "simulation from scratch", allowing self-driving environments to be created from arbitrary textual descriptions, images and sketches.