Predicting Model Failure using Saliency Maps in Autonomous Driving Systems
This work addresses the need for improved robustness and failure estimation in safety-critical industries like autonomous driving, though it appears incremental as it builds on existing saliency map techniques.
The paper tackles the problem of predicting failures in machine learning systems for safety-critical applications by training a student model to predict errors based on saliency maps, with preliminary results demonstrated on an autonomous vehicle steering control system.
While machine learning systems show high success rate in many complex tasks, research shows they can also fail in very unexpected situations. Rise of machine learning products in safety-critical industries cause an increase in attention in evaluating model robustness and estimating failure probability in machine learning systems. In this work, we propose a design to train a student model -- a failure predictor -- to predict the main model's error for input instances based on their saliency map. We implement and review the preliminary results of our failure predictor model on an autonomous vehicle steering control system as an example of safety-critical applications.