Efficient and Robust 2D-to-BEV Representation Learning via Geometry-guided Kernel Transformer
This addresses the need for efficient and robust perception in autonomous driving systems, particularly for real-time applications, though it appears incremental as it builds on existing 2D-to-BEV methods.
The paper tackles the problem of learning Bird's Eye View (BEV) representations from camera images for autonomous driving, achieving state-of-the-art real-time segmentation results with 38.0 mIoU on the nuScenes dataset and running at up to 72.3 FPS.
Learning Bird's Eye View (BEV) representation from surrounding-view cameras is of great importance for autonomous driving. In this work, we propose a Geometry-guided Kernel Transformer (GKT), a novel 2D-to-BEV representation learning mechanism. GKT leverages the geometric priors to guide the transformer to focus on discriminative regions and unfolds kernel features to generate BEV representation. For fast inference, we further introduce a look-up table (LUT) indexing method to get rid of the camera's calibrated parameters at runtime. GKT can run at $72.3$ FPS on 3090 GPU / $45.6$ FPS on 2080ti GPU and is robust to the camera deviation and the predefined BEV height. And GKT achieves the state-of-the-art real-time segmentation results, i.e., 38.0 mIoU (100m$\times$100m perception range at a 0.5m resolution) on the nuScenes val set. Given the efficiency, effectiveness, and robustness, GKT has great practical values in autopilot scenarios, especially for real-time running systems. Code and models will be available at \url{https://github.com/hustvl/GKT}.