Beyond RMSE: Do machine-learned models of road user interaction produce human-like behavior?
This addresses the need for more human-like behavior prediction in autonomous vehicles, though it is incremental as it proposes new evaluation metrics rather than a novel model.
The paper tackled the problem that machine-learned models for predicting road user behavior in autonomous vehicles rely too heavily on quantitative error metrics like RMSE, which may not capture human-like behavior. The result was an analysis showing that while models captured basic kinematic-dependent merging behavior, they struggled with nuanced courtesy lane changes and collision aversion, highlighting the inadequacy of current metrics.
Autonomous vehicles use a variety of sensors and machine-learned models to predict the behavior of surrounding road users. Most of the machine-learned models in the literature focus on quantitative error metrics like the root mean square error (RMSE) to learn and report their models' capabilities. This focus on quantitative error metrics tends to ignore the more important behavioral aspect of the models, raising the question of whether these models really predict human-like behavior. Thus, we propose to analyze the output of machine-learned models much like we would analyze human data in conventional behavioral research. We introduce quantitative metrics to demonstrate presence of three different behavioral phenomena in a naturalistic highway driving dataset: 1) The kinematics-dependence of who passes a merging point first 2) Lane change by an on-highway vehicle to accommodate an on-ramp vehicle 3) Lane changes by vehicles on the highway to avoid lead vehicle conflicts. Then, we analyze the behavior of three machine-learned models using the same metrics. Even though the models' RMSE value differed, all the models captured the kinematic-dependent merging behavior but struggled at varying degrees to capture the more nuanced courtesy lane change and highway lane change behavior. Additionally, the collision aversion analysis during lane changes showed that the models struggled to capture the physical aspect of human driving: leaving adequate gap between the vehicles. Thus, our analysis highlighted the inadequacy of simple quantitative metrics and the need to take a broader behavioral perspective when analyzing machine-learned models of human driving predictions.