Lift, Splat, Shoot: Encoding Images From Arbitrary Camera Rigs by Implicitly Unprojecting to 3D
This addresses the perception and fusion challenge for autonomous vehicles by providing a robust, end-to-end solution that handles arbitrary camera rigs and calibration errors, though it is incremental in improving existing bird's-eye-view methods.
The paper tackles the problem of generating a bird's-eye-view representation from multiple camera images for autonomous vehicles, achieving state-of-the-art performance on tasks like object and map segmentation by outperforming all baselines and prior work. It also demonstrates that this representation enables interpretable end-to-end motion planning by integrating trajectory templates into the output cost map.
The goal of perception for autonomous vehicles is to extract semantic representations from multiple sensors and fuse these representations into a single "bird's-eye-view" coordinate frame for consumption by motion planning. We propose a new end-to-end architecture that directly extracts a bird's-eye-view representation of a scene given image data from an arbitrary number of cameras. The core idea behind our approach is to "lift" each image individually into a frustum of features for each camera, then "splat" all frustums into a rasterized bird's-eye-view grid. By training on the entire camera rig, we provide evidence that our model is able to learn not only how to represent images but how to fuse predictions from all cameras into a single cohesive representation of the scene while being robust to calibration error. On standard bird's-eye-view tasks such as object segmentation and map segmentation, our model outperforms all baselines and prior work. In pursuit of the goal of learning dense representations for motion planning, we show that the representations inferred by our model enable interpretable end-to-end motion planning by "shooting" template trajectories into a bird's-eye-view cost map output by our network. We benchmark our approach against models that use oracle depth from lidar. Project page with code: https://nv-tlabs.github.io/lift-splat-shoot .