TransFuser: Imitation with Transformer-Based Sensor Fusion for Autonomous Driving
This addresses the challenge of robust end-to-end driving for autonomous vehicles in high-density traffic, representing a strong specific gain rather than a broad paradigm shift.
The paper tackles the problem of integrating complementary sensors for autonomous driving in complex scenarios with dense traffic, proposing TransFuser, a transformer-based sensor fusion method that outperforms prior work on the CARLA leaderboard and reduces collisions by 48% compared to geometry-based fusion.
How should we integrate representations from complementary sensors for autonomous driving? Geometry-based fusion has shown promise for perception (e.g. object detection, motion forecasting). However, in the context of end-to-end driving, we find that imitation learning based on existing sensor fusion methods underperforms in complex driving scenarios with a high density of dynamic agents. Therefore, we propose TransFuser, a mechanism to integrate image and LiDAR representations using self-attention. Our approach uses transformer modules at multiple resolutions to fuse perspective view and bird's eye view feature maps. We experimentally validate its efficacy on a challenging new benchmark with long routes and dense traffic, as well as the official leaderboard of the CARLA urban driving simulator. At the time of submission, TransFuser outperforms all prior work on the CARLA leaderboard in terms of driving score by a large margin. Compared to geometry-based fusion, TransFuser reduces the average collisions per kilometer by 48%.