Disengagement Analysis and Field Tests of a Prototypical Open-Source Level 4 Autonomous Driving System
This addresses the problem of assessing real-world robustness in open-source autonomous driving systems for researchers and developers, though it is incremental in applying existing disengagement analysis methods to a new context.
The study evaluated a prototypical open-source Level 4 autonomous driving system over 236 km of mixed traffic, finding a spatial disengagement rate of 0.127 per km, with perception and planning failures accounting for 40% and 26.7% of interventions, respectively.
Proprietary Autonomous Driving Systems are typically evaluated through disengagements, unplanned manual interventions to alter vehicle behavior, as annually reported by the California Department of Motor Vehicles. However, the real-world capabilities of prototypical open-source Level 4 vehicles over substantial distances remain largely unexplored. This study evaluates a research vehicle running an Autoware-based software stack across 236 km of mixed traffic. By classifying 30 disengagements across 26 rides with a novel five-level criticality framework, we observed a spatial disengagement rate of 0.127 1/km. Interventions predominantly occurred at lower speeds near static objects and traffic lights. Perception and Planning failures accounted for 40% and 26.7% of disengagements, respectively, largely due to object-tracking losses and operational deadlocks caused by parked vehicles. Frequent, unnecessary interventions highlighted a lack of trust on the part of the safety driver. These results show that while open-source software enables extensive operations, disengagement analysis is vital for uncovering robustness issues missed by standard metrics.