Super Agents and Confounders: Influence of surrounding agents on vehicle trajectory prediction
For autonomous driving researchers, this work identifies and addresses a critical flaw in trajectory prediction models, offering a simple fix that improves robustness.
The paper reveals that many surrounding agents degrade trajectory prediction accuracy in state-of-the-art models, and proposes a Conditional Information Bottleneck (CIB) that improves performance and robustness by selectively compressing agent features. Experiments show consistent gains across datasets and architectures.
In highly interactive driving scenes, trajectory prediction is conditioned on information from surrounding traffic participants such as cars and pedestrians. Our main contribution is a comprehensive analysis of state-of-the-art trajectory predictors, which reveals a surprising and critical flaw: many surrounding agents degrade prediction accuracy rather than improve it. Using Shapley-based attribution, we rigorously demonstrate that models learn unstable and non-causal decision-making schemes that vary significantly across training runs. Building on these insights, we propose to integrate a Conditional Information Bottleneck (CIB), which does not require additional supervision and is trained to effectively compress agent features as well as ignore those that are not beneficial for the prediction task. Comprehensive experiments using multiple datasets and model architectures demonstrate that this simple yet effective approach not only improves overall trajectory prediction performance in many cases but also increases robustness to different perturbations. Our results highlight the importance of selectively integrating contextual information, which can often contain spurious or misleading signals, in trajectory prediction. Moreover, we provide interpretable metrics for identifying non-robust behavior and present a promising avenue towards a solution.