Can I trust my anomaly detection system? A case study based on explainable AI
This work addresses trust issues in anomaly detection for users in fields like security or healthcare, but it is incremental as it applies existing explainable AI techniques to a known problem.
The study investigated the robustness of variational autoencoder-based anomaly detection systems using explainable AI methods, revealing that many samples are flagged as anomalous due to incorrect or misleading factors rather than genuine anomalies.
Generative models based on variational autoencoders are a popular technique for detecting anomalies in images in a semi-supervised context. A common approach employs the anomaly score to detect the presence of anomalies, and it is known to reach high level of accuracy on benchmark datasets. However, since anomaly scores are computed from reconstruction disparities, they often obscure the detection of various spurious features, raising concerns regarding their actual efficacy. This case study explores the robustness of an anomaly detection system based on variational autoencoder generative models through the use of eXplainable AI methods. The goal is to get a different perspective on the real performances of anomaly detectors that use reconstruction differences. In our case study we discovered that, in many cases, samples are detected as anomalous for the wrong or misleading factors.