WorldSplat: Gaussian-Centric Feed-Forward 4D Scene Generation for Autonomous Driving
This work addresses the need for scalable and controllable training data in autonomous driving by bridging the gap between scene generation and reconstruction, though it appears incremental as it builds on existing diffusion and Gaussian methods.
The paper tackles the problem of generating 4D driving scenes for autonomous driving by proposing WorldSplat, a feed-forward framework that produces high-fidelity, temporally and spatially consistent multi-track novel-view videos, as demonstrated through extensive experiments on benchmark datasets.
Recent advances in driving-scene generation and reconstruction have demonstrated significant potential for enhancing autonomous driving systems by producing scalable and controllable training data. Existing generation methods primarily focus on synthesizing diverse and high-fidelity driving videos; however, due to limited 3D consistency and sparse viewpoint coverage, they struggle to support convenient and high-quality novel-view synthesis (NVS). Conversely, recent 3D/4D reconstruction approaches have significantly improved NVS for real-world driving scenes, yet inherently lack generative capabilities. To overcome this dilemma between scene generation and reconstruction, we propose WorldSplat, a novel feed-forward framework for 4D driving-scene generation. Our approach effectively generates consistent multi-track videos through two key steps: (i) We introduce a 4D-aware latent diffusion model integrating multi-modal information to produce pixel-aligned 4D Gaussians in a feed-forward manner. (ii) Subsequently, we refine the novel view videos rendered from these Gaussians using a enhanced video diffusion model. Extensive experiments conducted on benchmark datasets demonstrate that WorldSplat effectively generates high-fidelity, temporally and spatially consistent multi-track novel view driving videos. Project: https://wm-research.github.io/worldsplat/