Learning to Generate 4D LiDAR Sequences
This work provides a foundation for LiDAR-based simulation and data augmentation, addressing an underexplored area in generative models for accurate 3D perception.
The paper tackles the problem of generating 4D LiDAR sequences from language, addressing challenges in controllability and temporal stability, and presents LiDARCrafter, which achieves state-of-the-art fidelity, controllability, and temporal consistency on nuScenes.
While generative world models have advanced video and occupancy-based data synthesis, LiDAR generation remains underexplored despite its importance for accurate 3D perception. Extending generation to 4D LiDAR data introduces challenges in controllability, temporal stability, and evaluation. We present LiDARCrafter, a unified framework that converts free-form language into editable LiDAR sequences. Instructions are parsed into ego-centric scene graphs, which a tri-branch diffusion model transforms into object layouts, trajectories, and shapes. A range-image diffusion model generates the initial scan, and an autoregressive module extends it into a temporally coherent sequence. The explicit layout design further supports object-level editing, such as insertion or relocation. To enable fair assessment, we provide EvalSuite, a benchmark spanning scene-, object-, and sequence-level metrics. On nuScenes, LiDARCrafter achieves state-of-the-art fidelity, controllability, and temporal consistency, offering a foundation for LiDAR-based simulation and data augmentation.