Localization Is All You Evaluate: Data Leakage in Online Mapping Datasets and How to Fix It
This addresses a critical data leakage problem for researchers and practitioners in autonomous driving and robotics, revealing inflated performance and misleading conclusions in online mapping, though it is incremental as it focuses on fixing dataset splits rather than introducing new methods.
The paper identifies data leakage in online mapping datasets like nuScenes and Argoverse 2, where over 80% and 40% of validation/test samples are near training samples, causing inflated performance by evaluating localization rather than generalization to unseen locations; it proposes geographically disjoint splits, showing methods drop over 45 mAP on proper splits and reassessing prior design choices yields diverging conclusions.
The task of online mapping is to predict a local map using current sensor observations, e.g. from lidar and camera, without relying on a pre-built map. State-of-the-art methods are based on supervised learning and are trained predominantly using two datasets: nuScenes and Argoverse 2. However, these datasets revisit the same geographic locations across training, validation, and test sets. Specifically, over $80$% of nuScenes and $40$% of Argoverse 2 validation and test samples are less than $5$ m from a training sample. At test time, the methods are thus evaluated more on how well they localize within a memorized implicit map built from the training data than on extrapolating to unseen locations. Naturally, this data leakage causes inflated performance numbers and we propose geographically disjoint data splits to reveal the true performance in unseen environments. Experimental results show that methods perform considerably worse, some dropping more than $45$ mAP, when trained and evaluated on proper data splits. Additionally, a reassessment of prior design choices reveals diverging conclusions from those based on the original split. Notably, the impact of lifting methods and the support from auxiliary tasks (e.g., depth supervision) on performance appears less substantial or follows a different trajectory than previously perceived. Splits can be found at https://github.com/LiljaAdam/geographical-splits