ForeSight: Multi-View Streaming Joint Object Detection and Trajectory Forecasting
This addresses the challenge of accurate 3D perception in autonomous driving by eliminating explicit object association, which is an incremental advance over existing multi-view detection and forecasting models.
The paper tackles the problem of joint object detection and trajectory forecasting for autonomous vehicles by introducing ForeSight, a framework that integrates these tasks to leverage temporal cues, resulting in state-of-the-art performance with a 54.9% EPA and a 9.3% improvement over previous methods on the nuScenes dataset.
We introduce ForeSight, a novel joint detection and forecasting framework for vision-based 3D perception in autonomous vehicles. Traditional approaches treat detection and forecasting as separate sequential tasks, limiting their ability to leverage temporal cues. ForeSight addresses this limitation with a multi-task streaming and bidirectional learning approach, allowing detection and forecasting to share query memory and propagate information seamlessly. The forecast-aware detection transformer enhances spatial reasoning by integrating trajectory predictions from a multiple hypothesis forecast memory queue, while the streaming forecast transformer improves temporal consistency using past forecasts and refined detections. Unlike tracking-based methods, ForeSight eliminates the need for explicit object association, reducing error propagation with a tracking-free model that efficiently scales across multi-frame sequences. Experiments on the nuScenes dataset show that ForeSight achieves state-of-the-art performance, achieving an EPA of 54.9%, surpassing previous methods by 9.3%, while also attaining the best mAP and minADE among multi-view detection and forecasting models.