ETA: Efficiency through Thinking Ahead, A Dual Approach to Self-Driving with Large Models
This addresses the challenge of timely response in self-driving systems by enabling efficient use of large models, representing an incremental improvement over existing dual-system architectures.
The paper tackles the problem of using large models in self-driving systems without sacrificing inference speed by proposing ETA, an asynchronous dual-system that shifts intensive computations to previous time steps and performs batch inference, achieving an 8% improvement in driving score to 69.53 on the Bench2Drive CARLA Leaderboard-v2 benchmark while maintaining near-real-time speed at 50 ms.
How can we benefit from large models without sacrificing inference speed, a common dilemma in self-driving systems? A prevalent solution is a dual-system architecture, employing a small model for rapid, reactive decisions and a larger model for slower but more informative analyses. Existing dual-system designs often implement parallel architectures where inference is either directly conducted using the large model at each current frame or retrieved from previously stored inference results. However, these works still struggle to enable large models for a timely response to every online frame. Our key insight is to shift intensive computations of the current frame to previous time steps and perform a batch inference of multiple time steps to make large models respond promptly to each time step. To achieve the shifting, we introduce Efficiency through Thinking Ahead (ETA), an asynchronous system designed to: (1) propagate informative features from the past to the current frame using future predictions from the large model, (2) extract current frame features using a small model for real-time responsiveness, and (3) integrate these dual features via an action mask mechanism that emphasizes action-critical image regions. Evaluated on the Bench2Drive CARLA Leaderboard-v2 benchmark, ETA advances state-of-the-art performance by 8% with a driving score of 69.53 while maintaining a near-real-time inference speed at 50 ms.