Street Gaussians without 3D Object Tracker
This work addresses the problem of robust scene reconstruction for autonomous driving systems by eliminating the need for manual labeling or unreliable 3D trackers, representing an incremental improvement over prior methods.
The paper tackles the challenge of reconstructing dynamic objects in driving scenes without relying on 3D object trackers, which often generalize poorly, by using 2D deep trackers and a motion learning strategy to correct errors, achieving superior performance on Waymo-NOTR and KITTI datasets.
Realistic scene reconstruction in driving scenarios poses significant challenges due to fast-moving objects. Most existing methods rely on labor-intensive manual labeling of object poses to reconstruct dynamic objects in canonical space and move them based on these poses during rendering. While some approaches attempt to use 3D object trackers to replace manual annotations, the limited generalization of 3D trackers -- caused by the scarcity of large-scale 3D datasets -- results in inferior reconstructions in real-world settings. In contrast, 2D foundation models demonstrate strong generalization capabilities. To eliminate the reliance on 3D trackers and enhance robustness across diverse environments, we propose a stable object tracking module by leveraging associations from 2D deep trackers within a 3D object fusion strategy. We address inevitable tracking errors by further introducing a motion learning strategy in an implicit feature space that autonomously corrects trajectory errors and recovers missed detections. Experimental results on Waymo-NOTR and KITTI show that our method outperforms existing approaches. Our code will be released on https://lolrudy.github.io/No3DTrackSG/.