Enhanced Motion Forecasting with Plug-and-Play Multimodal Large Language Models
This work addresses the problem of cost-effective generalization in autonomous driving motion forecasting, though it is incremental as it builds on existing models with MLLM augmentation.
The paper tackles the challenge of generalizing motion forecasting in autonomous driving to diverse real-world scenarios by proposing Plug-and-Forecast (PnF), a plug-and-play approach that augments existing models with multimodal large language models (MLLMs) to extract structured scene understanding via natural language prompts. The method achieves consistent performance improvements on the Waymo Open Motion and nuScenes datasets without requiring fine-tuning.
Current autonomous driving systems rely on specialized models for perceiving and predicting motion, which demonstrate reliable performance in standard conditions. However, generalizing cost-effectively to diverse real-world scenarios remains a significant challenge. To address this, we propose Plug-and-Forecast (PnF), a plug-and-play approach that augments existing motion forecasting models with multimodal large language models (MLLMs). PnF builds on the insight that natural language provides a more effective way to describe and handle complex scenarios, enabling quick adaptation to targeted behaviors. We design prompts to extract structured scene understanding from MLLMs and distill this information into learnable embeddings to augment existing behavior prediction models. Our method leverages the zero-shot reasoning capabilities of MLLMs to achieve significant improvements in motion prediction performance, while requiring no fine-tuning -- making it practical to adopt. We validate our approach on two state-of-the-art motion forecasting models using the Waymo Open Motion Dataset and the nuScenes Dataset, demonstrating consistent performance improvements across both benchmarks.