Test Automation with Grad-CAM Heatmaps -- A Future Pipe Segment in MLOps for Vision AI?
This addresses the problem of model interpretability for critical applications like automotive systems, but it is incremental as it applies an existing method to a new context.
The paper tackles the opaqueness of deep neural networks in computer vision by using Grad-CAM heatmaps to increase explainability for an image recognition model trained for a pedestrian underpass, proposing automated heatmap analysis as an MLOps pipeline segment to detect biased models based on invalid pixel activations.
Machine Learning (ML) is a fundamental part of modern perception systems. In the last decade, the performance of computer vision using trained deep neural networks has outperformed previous approaches based on careful feature engineering. However, the opaqueness of large ML models is a substantial impediment for critical applications such as in the automotive context. As a remedy, Gradient-weighted Class Activation Mapping (Grad-CAM) has been proposed to provide visual explanations of model internals. In this paper, we demonstrate how Grad-CAM heatmaps can be used to increase the explainability of an image recognition model trained for a pedestrian underpass. We argue how the heatmaps support compliance to the EU's seven key requirements for Trustworthy AI. Finally, we propose adding automated heatmap analysis as a pipe segment in an MLOps pipeline. We believe that such a building block can be used to automatically detect if a trained ML-model is activated based on invalid pixels in test images, suggesting biased models.