Visual Point Cloud Forecasting enables Scalable Autonomous Driving
This addresses the problem of scalable autonomous driving for developers and researchers by providing a novel pre-training approach, though it is incremental in advancing existing methods for this domain.
The paper tackles the lack of pre-training methods for scalable visual autonomous driving by introducing visual point cloud forecasting, a task that predicts future point clouds from historical visual input, resulting in gains such as 3.1% NDS on 3D detection and ~15% less collision rate on planning.
In contrast to extensive studies on general vision, pre-training for scalable visual autonomous driving remains seldom explored. Visual autonomous driving applications require features encompassing semantics, 3D geometry, and temporal information simultaneously for joint perception, prediction, and planning, posing dramatic challenges for pre-training. To resolve this, we bring up a new pre-training task termed as visual point cloud forecasting - predicting future point clouds from historical visual input. The key merit of this task captures the synergic learning of semantics, 3D structures, and temporal dynamics. Hence it shows superiority in various downstream tasks. To cope with this new problem, we present ViDAR, a general model to pre-train downstream visual encoders. It first extracts historical embeddings by the encoder. These representations are then transformed to 3D geometric space via a novel Latent Rendering operator for future point cloud prediction. Experiments show significant gain in downstream tasks, e.g., 3.1% NDS on 3D detection, ~10% error reduction on motion forecasting, and ~15% less collision rate on planning.