Realistic and Controllable 3D Gaussian-Guided Object Editing for Driving Video Generation
This addresses the need for cost-effective and safe generation of corner cases for autonomous driving systems, though it is incremental as it builds on existing 3D Gaussian and generative model approaches.
The paper tackles the problem of generating diverse driving scenarios by editing objects in captured sensor data, proposing G^2Editor to achieve photorealistic and precise object editing with improved pose controllability and visual quality, as demonstrated on the Waymo Open Dataset.
Corner cases are crucial for training and validating autonomous driving systems, yet collecting them from the real world is often costly and hazardous. Editing objects within captured sensor data offers an effective alternative for generating diverse scenarios, commonly achieved through 3D Gaussian Splatting or image generative models. However, these approaches often suffer from limited visual fidelity or imprecise pose control. To address these issues, we propose G^2Editor, a framework designed for photorealistic and precise object editing in driving videos. Our method leverages a 3D Gaussian representation of the edited object as a dense prior, injected into the denoising process to ensure accurate pose control and spatial consistency. A scene-level 3D bounding box layout is employed to reconstruct occluded areas of non-target objects. Furthermore, to guide the appearance details of the edited object, we incorporate hierarchical fine-grained features as additional conditions during generation. Experiments on the Waymo Open Dataset demonstrate that G^2Editor effectively supports object repositioning, insertion, and deletion within a unified framework, outperforming existing methods in both pose controllability and visual quality, while also benefiting downstream data-driven tasks.