Metrics for the Evaluation of localisation Robustness
This work addresses the lack of robustness evaluation in localization systems for self-driving vehicles, which is crucial for real-world safety but often overlooked in incremental algorithm development.
The paper tackles the problem of evaluating the robustness of localization systems in autonomous vehicles, proposing novel metrics that can quantify robustness with or without accurate ground truth, and demonstrates their application through experimental analysis on well-known localization strategies.
Robustness and safety are crucial properties for the real-world application of autonomous vehicles. One of the most critical components of any autonomous system is localisation. During the last 20 years there has been significant progress in this area with the introduction of very efficient algorithms for mapping, localisation and SLAM. Many of these algorithms present impressive demonstrations for a particular domain, but fail to operate reliably with changes to the operating environment. The aspect of robustness has not received enough attention and localisation systems for self-driving vehicle applications are seldom evaluated for their robustness. In this paper we propose novel metrics to effectively quantify localisation robustness with or without an accurate ground truth. The experimental results present a comprehensive analysis of the application of these metrics against a number of well known localisation strategies.